Announcement

Collapse
No announcement yet.

The Unreal Marketplace Improvement and Feedback Thread

Collapse
This is a sticky topic.
X
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • replied
    Sorry about that! It's up now: https://trello.com/c/1DA789CK/333-su...d-starter-pack

    Time to advertise and get votes! Tell your friends, your social networks, the forums, and the world! You have a lot of presence here on the forums and it might not be a bad idea to add it to your signature.

    Leave a comment:


  • replied
    Originally posted by Jon Jones View Post
    That's a good idea, actually! We're working on trying to communicate more frequently, and we're bringing on another person next week to help us with the customer-facing side of the Marketplace.

    Thinking about it, I don't think I've clearly described the submission process in a straight line before. Here's how it works:

    1) Content creator submits through the submission form
    2) Epic reviews the screenshots, video, and description of the content
    3) Epic decides: Yes, No, or Request Changes
    4) When Epic says 'Yes' we request the files
    5) Internal QA on the files
    6) Epic decides: Yes, No, or Request Changes
    7) When Epic says 'Yes' then the content goes to public Trello voting
    8) Community decides with voting: Yes or No
    9) When the voting period completes, Epic confirms the files and description are up to date, and schedules release
    10) Done!

    The time-consuming parts are step 3, 6, and 8. Lots of feedback and iteration happen at step 3 and 6 to bring it up to Epic's quality standards. When we request specific changes, that turns into a feedback loop to bring it up to quality before proceeding. In general, when you get specific feedback from us on a submission at that stage, that usually means you can turn a 'no' into a 'yes.' Unfortunately sometimes that's not always possible, and some submissions just aren't up to the quality that the Marketplace needs, and a firm 'no' is all we can do. As a rule, though, we want people to succeed and we help how we can, within reason.

    I just checked your submission and we finished QAing it on Tuesday night, and it'll be up on the public Trello boards for voting within the next hour. You'll get an email from us.

    BTW, seriously cool pack and example map!
    Thanks, I glad you like it! And breakdown is extremely helpful, however my submission is still not on Trello and I didn't get an email

    Leave a comment:


  • replied
    Thanks for the breakdown, a tracking bar of sorts like zeOrb suggested would be pretty cool. Then again, I always spam my refresh button on those pages so it might be just another thing to prevent me from getting things done. =P

    Leave a comment:


  • replied
    That's a good idea, actually! We're working on trying to communicate more frequently, and we're bringing on another person next week to help us with the customer-facing side of the Marketplace.

    Thinking about it, I don't think I've clearly described the submission process in a straight line before. Here's how it works:

    1) Content creator submits through the submission form
    2) Epic reviews the screenshots, video, and description of the content
    3) Epic decides: Yes, No, or Request Changes
    4) When Epic says 'Yes' we request the files
    5) Internal QA on the files
    6) Epic decides: Yes, No, or Request Changes
    7) When Epic says 'Yes' then the content goes to public Trello voting
    8) Community decides with voting: Yes or No
    9) When the voting period completes, Epic confirms the files and description are up to date, and schedules release
    10) Done!

    The time-consuming parts are step 3, 6, and 8. Lots of feedback and iteration happen at step 3 and 6 to bring it up to Epic's quality standards. When we request specific changes, that turns into a feedback loop to bring it up to quality before proceeding. In general, when you get specific feedback from us on a submission at that stage, that usually means you can turn a 'no' into a 'yes.' Unfortunately sometimes that's not always possible, and some submissions just aren't up to the quality that the Marketplace needs, and a firm 'no' is all we can do. As a rule, though, we want people to succeed and we help how we can, within reason.

    I just checked your submission and we finished QAing it on Tuesday night, and it'll be up on the public Trello boards for voting within the next hour. You'll get an email from us.

    BTW, seriously cool pack and example map!
    Last edited by Jon Jones; 04-16-2015, 01:02 PM.

    Leave a comment:


  • replied
    It would be cool if we could see status of submission from the beginning to the very end.
    I've sent project files a week ago and did not got any reply yet, I've heard there are 27 submissions for QA, so I'm pretty sure it just waiting for its turn, but there is no any indication of that visible to the submitter.
    I think it would only benefit if you could check submission status somewhere, like tracking package and etc

    Leave a comment:


  • replied
    I see, thanks for the clarification on that. =)

    Leave a comment:


  • replied
    Technically we are, but that's behind the veil a bit. After this month, we'll be more publicly obvious about it. Since GDC we're recalibrating what the voting \ fast-tracking threshold should be, so we're letting them stay up a little longer than usual. However, once a submission gets past a certain number of votes we keep it up on the board, and start working behind the scenes to get the latest files and info to prep and test it, then schedule it for release. For our release on Wednesday, we currently have 9 packs planned for release, and I'm working on bumping that number up each week. I'm actually heading into one of our weekly content review meetings in 15 to review the latest batch of submissions and to start adding more content to our upcoming releases.

    Leave a comment:


  • replied
    Are you guys still doing the timed releases? I thought you moved on to releasing as soon as it gained sufficient votes on trello now since people were supportive of that approach as opposed to holding on to it for a while?

    Leave a comment:


  • replied
    I think Jon should still do the nice weekly blog posts about new stuff which was added the last week. So if you are interested, you can read it and see what's new and you won't miss anything.

    We don't talk about "unfinished work" here, but about when finished and "approved by epic" products should appear on the marketplace. Whether Epic should wait for people to vote on or Epic should just release them when they are ready.

    Leave a comment:


  • replied
    Originally posted by John Alcatraz View Post
    Why do you wan't to care about "community input"? If I want to sell a book on amazon, I just upload it, manage the financial stuff and then people can buy it! If "community interest" is low, well, then just no one will buy it. Will only hurt the seller, not the community.

    Just put everything on the marketplace when it's ready, this means when you are satisfied with the quality and have successfully managed all the financial things with the seller. Then just put it on the marketplace! Immediately, not bundled every week! If it's ready at Monday 10AM, just put it on the marketplace on Monday 10AM. If it's ready at Thursday 12AM, put it on the marketplace on Thursday 12AM!
    The Trello system is in place to allow the community to prioritize assets on a release schedule and to placate the community during this time of growth and transition. The past few weeks have been killer on workload and process changes for Epic's marketplace/QA team (I'm assuming). There's also the point of managing flow through the marketplace. Even Apple has an organically regulated system on the App store - sure you can release your App whenever you want but they update the front page on a certain day each week, which means most people wait until the day before to post their product to have the best chances for front page coverage. This ensures some sort of continuity for buyers and keeps the market alive and growing at a steady pace rather than having roller-coaster days, where one day there's 15 submissions and another day there's 0. As a buyer I'd feel a little overwhelmed if random assets just popped up each day - I'd surely miss some. As a buyer on the app store, I check for top new apps on Tuesdays.

    I don't entirely understand where all the heat is coming from - use the forums to gauge interest and get feedback as you work on your product. We're really good at giving feedback. Once the product is ready to sell, submit it for Epic's review and Trello posting. Unreal Marketplace aside, why would you ever submit unfinished work for potential sale? It doesn't make sense.

    Leave a comment:


  • replied
    I said put it on the marketplace when it's through Epics QA and the financial stuff is managed. Epic will not let trash or mediocre content through their QA, so what you talk about could never happen.

    Leave a comment:


  • replied
    Originally posted by John Alcatraz View Post
    Why do you wan't to care about "community input"? If I want to sell a book on amazon, I just upload it, manage the financial stuff and then people can buy it! If "community interest" is low, well, then just no one will buy it. Will only hurt the seller, not the community.

    Just put everything on the marketplace when it's ready, this means when you are satisfied with the quality and have successfully managed all the financial things with the seller. Then just put it on the marketplace! Immediately, not bundled every week! If it's ready at Monday 10AM, just put it on the marketplace on Monday 10AM. If it's ready at Thursday 12AM, put it on the marketplace on Thursday 12AM!
    I'm strongly disagree.
    If you put everything that people send to Marketplace immediately it will take only couple of weeks to turn Marketplace into trashcan. It will be filled with mediocre content that no one wants and it will hurt sales of good content, because visibility will be lower.
    I'm not saying it's perfect now, I think current rate of new items could be improved. Two release days per week may be better, but your proposal will simply decrease overall quality of content and decrease visibility for good stuff. Also it force marketplace team to process items faster what either requires more people to hire or will decrease quality of work.

    Leave a comment:


  • replied
    Why do you wan't to care about "community input"? If I want to sell a book on amazon, I just upload it, manage the financial stuff and then people can buy it! If "community interest" is low, well, then just no one will buy it. Will only hurt the seller, not the community.

    Just put everything on the marketplace when it's ready, this means when you are satisfied with the quality and have successfully managed all the financial things with the seller. Then just put it on the marketplace! Immediately, not bundled every week! If it's ready at Monday 10AM, just put it on the marketplace on Monday 10AM. If it's ready at Thursday 12AM, put it on the marketplace on Thursday 12AM!

    Leave a comment:


  • replied
    Originally posted by Osok View Post
    I am a bit curious about the trello voting. What is the threshold for 'fast tracking' items to the marketplace?
    For example, this one: advanced-dissolve-fx has 100 votes and has been on the trello for 4 weeks; yet it still hasn't been approved (it will need to wait the entire month? And then wait until it gets slotted into a release week?)
    The rough threshold for fast-tracking used to be roughly 30 votes, but it depends on the category. But since GDC and UE4 going free, that changed substantially. Before GDC it would take around a month to get 30 votes, and after GDC you can get that in a day or two. Right now we're finding out what the new threshold should be, and if there's going to be a dropoff in votes after the GDC bump.

    Now, here's the behind-the-scenes bit: fast-tracking happens around 30 votes, where we privately email the submitter and start getting the legal and financial paperwork signed. This can be very time-consuming, taking from days to weeks. So that happens while voting is still going! At this point we also encourage them to start advertising and getting more votes and bumping up interest in advance of the release.

    Now that files are required to be submitted for QA by EPIC before even reaching the trello, why have the trello at all? If assets are deemed of high enough quality by EPIC internal why wait any longer?
    Building public interest, working with the community, and advertising. My idealized vision for the Marketplace is to help turn it into an incubator for entrepreneurs so people can learn how to build an advertising campaign, drum up community support, and to apply effort within that voting time window so their release can really pop. I work behind the scenes with sellers to encourage this. We're still trying to improve this system, and it's entirely possible that Trello may not be the best way to keep doing it.

    You mentioned 27 assets passed the internal QA bar you've set. If we continue the pace of about 5 assets per week some of that stuff won't see the light of day until the middle of May even though it is ready NOW.

    I guess what I am saying is I can not think of any legitimate reason for the wait.
    Actually, they're *far* from ready at that point. Just because a submission comes in doesn't mean it's ready to publish immediately. As I said above, the legal and financial paperwork for new sellers can take from days to weeks. And when we do QA on the packs and find problems with the content and request changes, that also takes days to weeks to fix. Sometimes it happens quickly, and sometimes it's slow. It depends on the nature of the feedback we send and how long it takes to set up and fix. I know that people have day jobs and work on Marketplace content when they have time, so real life is definitely a factor.

    And in many cases, submitters simply don't respond even after multiple emails. There are probably 75+ submissions I can think of where I had to email submitters 2 to 4 times over the course of a month or two just to get them to respond, even if I've requested their files, or even if we're ready to publish. One submitter with multiple packs that we were ready to publish accidentally filtered our emails into a folder he never checked, and we had to chase them for two months before we figured out what happened. In cases like this, even if we're ready to publish the content immediately, there's really nothing I can do but keep emailing them and hoping they'll respond.

    Here's an important, behind-the-scenes look at QA: We do read the forums, and we get emails and PMs about different sellers' content and bugs that are in them. We also frequently update submitted content for bug fixes and to add new features. Each time this happens, we get a better idea of common issues and bugs to look for and test in our QA process, so we can avoid releasing content that has problems like this. This is precisely why we do QA before Trello and work with submitters to fix these issues before it goes to public voting.

    We've had cases where we've realized too late that some content is simply broken and not able to be fixed in time for the planned release, if ever. But if the public has already seen it and voted on it, whatever we decide to do is probably going to disappoint someone. Our choices are basically 1) reject it after people were looking forward to buying it, 2) delay it indefinitely at the last minute until the submitter can fix it, or 3) allow the release of something sub-par that's going to upset paying customers.

    So what looks like a slower intake right now is actually saving a substantial amount of time later for developers buying Marketplace content. We definitely take a hit on the perception of being slow to publish content, but ultimately I think the best thing to do in order to ensure quality. So we have to aggressively seek every way to speed up the process that we can. And we do, and we're always trying to get better and release more, as long as we can maintain quality.

    I'm sure the majority of users don't even know about the trello voting. I know epic wants to grow into a big marketplace like Unity Asset Store, but wants it to be of higher quality (justifiably so). The internal QA check by EPIC filters this to ensure quality so I think it is time to open up the flood gates Jon. If we limit ourselves to about 5 assets a week making the marketplace, we're limiting content creators, content consumers, and UE4 projects in general.
    I do agree on that point. In the coming weeks I'm wanting to push the weekly release up toward 10 releases each week to start. We're also honestly thinking about whether or not Trello is still the best way to handle public input on the Marketplace. It does technically add up to four weeks to the submission time for each asset. If the community becomes satisfied enough with the QA process Epic goes through and the approval process for choosing what's published, maybe we could find another way to get community input. The main balance we're trying to find there is that if we ever do move away from Trello, we want to maintain community involvement in a way that is honest and really means something. And we're always open to listening.

    I really appreciate you taking the time to give your feedback, and that goes for everyone else here. If you didn't care, you wouldn't say anything at all, and we want to return that respect.

    Leave a comment:


  • replied
    Osok, I completely agree with you! Epic should remove the annoying trello voting process. It does not make sense any longer. I also agree with you that most people don't even know trello. I know it, but I have never voted for anything there. Still, I will buy interesting stuff on the marketplace.

    Leave a comment:

Working...
X