Friday, January 27, 2012

Follow-Up to World of Workcraft

Here is the accompanying report we had to go along with our scenario of World of Workcraft. This report outlines some of the technical and research challenges. As before, credit should be shared with Rick Wash and Michael Bernstein. Blame should go to me.

Introduction
We envision a future comprised of a highly flexible workforce, where people with appropriate skill sets can be quickly and easily brought together on demand into flash teams to complete projects of varying sizes, difficulties, and time scales. These projects will also function as apprenticeships for people, helping workers to continually learn useful skills that will enable them to actively participate and adapt to changes in the dynamic knowledge economies of the twenty-first century.

We are starting to see early glimpses of this kind of future work force. For example, many graphic designers and programmers are hired on a consulting basis, working with short-term project teams that are dissolved once the project is completed. There are also numerous temporary work agencies that place workers into a wide range of positions. For example, Manpower Inc. is an agency for placing temporary workers, claiming over 400,000 clients per year. Finally, we are seeing increases in the number of people who telecommute, with over 17.2 million people occasionally telecommuting in 2008. As more and more media is digitized, and as the quality of collaboration tools increase, the number of people who can potentially work from anywhere in the United States will only increase.

Our vision is to vastly expand the number of domains and the number of people who can participate in this kind of knowledge economy. However, there are currently numerous technical, legal, and economic barriers in place, introducing transaction costs that create significant frictions and inefficiencies. Below, we outline several technical challenges that must be overcome before this vision can come to fruition.

There are many societal benefits if we achieve this vision. One is a flexible workforce that is continually adapting to changes in the world economy. Another benefit is far lower barriers for finding work that matches their skills and interests. A third benefit is greater innovation, as people can try many more ideas quickly and cheaply. People can also share knowledge and skills to more people. Furthermore, just as cloud computing vastly reduces one's front-end capital costs and space requirements and thus lower initial startup costs for new companies, crowd computing offers the same for the personnel needed for startups. A fourth benefit is a reduced need to commute to a central workplace, helping to reduce traffic congestion and pollution as well as improving quality of life.

Electronically Mediated Skilled Worker Marketplace
One of the major technical challenges we face in bringing forth this vision is the creation of an electronic marketplace for skilled labor and flash teams. Currently, finding and hiring skilled labor has high transaction costs: finding skilled workers, negotiating legal and HR contracts, and putting together effective teams.

We envision a new form of Internet-based electronically mediated work marketplace that can dramatically reduce the transaction costs of short-term skilled labor by efficiently matching skilled workers with market needs. The first challenge is specification: potential workers can specify their skills and their interests, and employers need to specify their needs and the projects they want completed. Currently on systems, this specification is usually very vague: specifics might be negotiated over time between employers and employees. In current microtask markets like Amazon Mechanical Turk, specifics might be left out entirely. As we push toward smaller tasks and automated matching, we need to make these specfications more specific. A worker isn't just "a programmer" but rather, someone with active skills in Python, Perl, rusty skills in Java, and recent experience with web development for Facebook apps. With sensors and computer vision technologies, these skills could also include those outside the computer, including for example mechanical skills or knowledge of a geographic area. Additionally, we need to have a way of warranting, or attesting to the validity of these skills.

Another related challenge for the electronic marketplace is efficient task-worker matching. Currently, most matching is done via an expensive search process: workers linearly scan through an enormous list of jobs looking for ones they might be interested, and then they apply for those jobs. (See for example, Chilton, L., et al. Task Search in a Human Computation Market). This search is a very inefficient method of matching skilled workers to employer needs. Instead, we propose centralizing the matching process and having computers assist the process of forming flash teams. But, what is the best role for computers to take, and how do we do this? One example where this has been successful is the National Resident Matching Program, where newly minted MDs go to get their first residency job after medical school. This is a centralized match where MDs submit where they would liek to work, and hospitals submit who they want to hire, and then the NRMP tells everyone who will work where.

When both workers and tasks are much more heterogeneous -- creating an entire job market -- we will need new sociotechnical designs to facilitate matching. This intelligence could incorporate elements of economics and computer science. Economics expertise can help ensure that the resulting matches are efficient and suggest appropriate incentives for workers and employers to honestly reveal their skills and needs. Computer science will be needed to help people specify their skills and tasks (HCI) and to build this system so that it can appropriately scale. Workers need a way of (semi-automatically) specifying their resume and warranting their skills and experience, but still having the flexibility to retain agency. Employers will need a way of clearly specifying job parameters, including specifying the work that needs to be done and any additional contracts -- financial, legal, intellectual property, benefits, etc. -- that are associated with the job.

The biggest challenge will be developing appropriate algorithms for doing this matching. There are both strong technical and economic constraints on this algorithm. The match is of high dimensionality; workers need to be matched to jobs across multiple skillsets, location constraints, financial/legal constraints, worker interests, task domains, and learning/development/long-term goals. Understanding and specifing the tradeoffs inherent in this type of match is very difficult, and using a computational algorithm -- such as a maching learning algorithm or other AI system for creating this matching -- is an open research problem. Additionally, there are considerable economic constraints on this algorithm. The resulting matches have to be satisfactory to both the workers and the employers. And the process -- the way the algorithm creates the match -- should be seen as fair, working best when both sides of the market are honest. This economic constraint, inducing honest revelation, is really important because if people are deceptive in their resume or their job descriptions, then the computer will not have the information it needs to effectively create matches.

Lifelong Learning
We hope that this type of system will support lifelong learning among the workers in the system. Being an ad-hoc worker in a system like this allows people try and do many small tasks quickly. Workers who want to learn new skills can work on projects outside their current skillset but within their zone of proximal development. One of the technical challenges for this system is to figure out how to specify the zone of proximal development and incorporate it into the matching process. We want people to receive matches near their current skill level so that they can learn. Alternatively, workers might be matched with a mentor who can oversee their work and help them learn a new domain. However, including these learning goals in a matching algorithm means that, to some extent, dishonest workers might be able to game the system and use it to get tasks that they should never be matched with. Allowing for mentorship and training and development should be supported, while simultaneously preventing people from gaming the system.

A second technical challenge incorporating learning into explicit feedback systems. Consider, for example, a reputation system like eBay's: a worker's early attempts may not be very skilled, but over time and with learning they will improve. However, if the reputation system still includes that early work as an important part of their formally specified reputation, that may artifically hold the worker down and prevent them from receiving appropriate work. We need to better design reputation and resume specification to allow for learning and change over time; a person's skills and reputation is not static but is constantly changing. (However, spammers may take advantage of such a system to appear more reputable than they actually are.)

A third technical challenge is helping workers figure out their zone of proximal development, and suggesting what types of "stretch" tasks might be the best way to expand skills and continue lifelong learning. The marketplace might be able to use a recommendation system to suggest new skills and projects to undertake to support lifelong learning: "other workers like you undertook these projects to develop their skills." However, those recommendations should include both what they are likely to agree to learn, but also include information about the market and what kinds of skills are needed and valuable.

Worker Skill Specification
To faciliate effective matching between workers and tasks, the system needs a strong reputation and skill system. Traditional resumes are concise but may not help with matching workers to tasks because they are fairly high-level. We envision a much richer range of skills in our system, enabling workers to fluidly move between tasks that exercise multiple aspects of their skill sets. Workers can build up an automatic resume of work history and skills: this includes traditional skills like programming in Python, as well as situational skills like living in Boston, hobbies like beer brewing, or familiarity with the Appalachian Trail, all of which could be useful depending on the specific task on hand. Potentially, managers could provide feedback to workers in ways that build up expertise profiles (or levels). Workers can then apply for more lucrative tasks that require high-level workers, like a "Level 5" Python programmer. This framework would also enable workers to take qualification and training tasks to "level up" and learn new skills, opening up new jobs and making re-training in a difficult economy much more feasible.

In technical terms, this goal requires a framework for workers to explicitly author and manage their skill set, as well as for an automatic reputation system like eBay or Amazon. Other challenges include accurately capturing what skills a person has, verifying those skills, and helping workers manage what skills are disclosed to others. These are all challenging from a sociotechnical design perspective, as workers' reputations are on the line.

Better Collaboration Workspaces
Workers and requesters require new interfaces to support complex, ad-hoc work structures. How can requesters (managers) control the process and ensure good results? How can workers learn more about the task they are performing, communicate with the requester, or even rewire the entire workflow? In this section we describe some research challenges for interfaces supporting workers and managers.

Managers need tools for task specification, awareness, and coordination of their distributed flash teams. They need to author workflows. Authoring should be as easy as describing the job to another person: "I need these ten interface designs mocked up in a vector art program. Then get another designer to iterate on them, and fifteen potential users to try each. I want a usability researcher to run that study and report the outcome to me." The manager can convey this request to a meta-worker or an algorithm, which transforms it into a workflow, or the manager might use a template workflow for specific kinds of tasks. Once the task has begun, the manager needs awareness and coordination support for the team. Can the manager inspect progress and ask workers at particular stages how the work is going, ensure effective communication flow between stages, and do quality control checks?

Workers need interface support to ensure that they complete the correct task and do it effectively. Currently, workers on systems like Mechanical Turk see only a thin slice of the problem, so it is very easy to misinterpret instructions. Workers need to be able to inspect the history of an artifact they are working on: who worked on this job previously, and what were they supposed to do? They need to be able to communicate with the manager, getting clarification or asking for in-progress feedback. They should also be able to suggest alternate workflows: what if the user experience engineer sees an early usability problem and wants to send the mockup back to the graphic designer for a quick fix, or wants to consult an accessibility expert before making a recommendation?

The Future of Work, or The World of Workcraft

A while back in 2010, I attended a CCC/CRA workshop in Ultra-Large-Scale Interaction. The goal of the workshop was to set out a broad research agenda for having thousands of people collaborate together on hard problems. A lot of the discussion focused on crowdsourcing, and how to apply crowdsourcing techniques to enable new kinds of work in the future.

Below is a scenario that Rick Wash, Michael Bernstein, and I put together for our vision of how crowdsourcing techniques for skilled workers might pan out in the future. We jokingly called it the World of Workcraft, since we drew some of our ideas from that popular game.

For example, in WoW, the game records your skill level as well as achievements demonstrating that you have some level of mastery. The game also has ways to facilitate assembling people based on their skills (e.g. tank, damage, healer), having these people operate in short-term groups to achieve specific objectives. We felt that some of these same ideas might be applicable for work in the future, where you might be able to quickly assemble groups of skilled people (rather than unskilled ones, as in MTurk), thus reducing the friction and transaction costs it takes to get things done as well as to find people with the right skill sets.

I've had our scenario lying around on my hard drive for a while, I figured it was about time to share it. I think that there are a lot of compelling ideas here from a broader perspective as well. We're starting to see tectonic shifts in how work is being done, in the form of more powerful technologies as well as a global workforce.

If we rewind back to the Industrial Revolution, we saw many major (and often painful) shifts as we transitioned from an agrarian society to an industrial one, including such things as urbanization, compulsory education, unions, social safety nets, feminism, and more. The changes we are now seeing as we continue accelerating towards an information society are likely just the tip of the iceberg, and will likely be just as significant (and painful) as the ones in the past.

At any rate, our scenario offers one glimpse at a possible future of what work might be like in the future. Incidentally, the two people running the workshop were Dan Olsen and Beth Mynatt, hence the names of our characters.

---------

Dan Mynatt has an idea for a technology start-up company, but he has training only in user interface design. He does not have the skills or manpower yet to staff an entire start-up: he wants to get his design ideas off the ground to show to some potential funders. This would typically take him weeks and considerable seed money, but Dan wants to get a refined mockup of his idea in one week and a limited budget. Dan's main skill is concept designing and storyboarding, so he begins by sketching lots of concept designs for his idea. He selects ten concepts that he likes. Normally he would spend weeks refining these interfaces and getting feedback, but with our system he can parallelize and recruit skilled workers for small design, implementation and evaluation tasks.

He logs on to our proposed task market to gather a flash team and workflow -- he wants to iterate on the designs, produce high-fidelity mockups and get user feedback. He uses a previous workflow template to specify the job. First, the designs are fed out to other designers. These designers give him feedback on the concepts -- professional crits and suggestions -- which he uses to revise the designs and cull down to seven to continue exploring. Within a few hours, he has now gotten the same amount of feedback he would have needed to wait hours or days to produce before.

Dan takes these seven designs and asks graphic designers to produce pixel-perfect mockups of them. He outsources a first usability study by recruiting a professional user testing engineer. The lead, Beth Olson, begins by developing metrics and tasks. She delegates each of these tasks to other user testing engineers, who in turn crowdsource the evaluation by asking unskilled users to run through the interface design alternatives.

For a moment, let's consider the perspective of one of the usability engineers, Judy Ackerman. She receives one of the high-fidelity mockups, as well as instructions to test it on at least ten users. She begins to play with the prototype himself, and discovers an obvious flaw in the design. Judy raises a flag to Dan, the manager, saying that the design needs a slight tweak before he can study it. Dan and Judy enter a conversation to refine the prototype, and Dan agrees to send it back to the mockup author for an edit. Finally, it comes back to Judy. She flips into the manager interface herself to author a user study workflow and get the requested feedback.

All this feedback makes its way back to Dan. Dan weeds out a few of the designs and produces some new alternatives: he is now down to three designs. Now he decides that it is time to build interactive prototypes and deploy them for feedback. First, Dan orchestrates a mini-development team to create interactive mockups for each of the designs. These teams include a graphic artist, at least one programmer, and a user testing engineer. This team builds and deploys a prototype, with the user testing engineer heading the evaluation.

Finally, given the feedback and prototypes, Dan selects a design recommendation. He tweaks the design, writes up documentation and finds a development team to help him execute the concept.

Why is life better?
  • Can test many more alternatives (10 product ideas in the time I could have done 1 internally)
  • Higher quality and lower risk (e.g. Threadless) because you get early feedback
  • By incorporating lots of people at each stage, you get varied feedback and multiple perspectives at critical points.
  • Supplement the shortcomings that come with a specialized skill set -- a designer is poor at coding and evaluation, but can ask for help in those situations.
  • Empowering people who don't have any of these skills, e.g. a concept person who knows a pain point but doesn't know anything about programming or designing or (perhaps) project management