AI & Web3 Video: Tools, Trends & Resources for the Creator Economy
An evolving research report initially published in January 2023
Research compiled by: The Gallery DAO / Art Official Intelligence LTD
Main contributors: @originstory @marianaconstantin @w0xt3r
OVERVIEW
The goal of this research was to unpack the tools, platforms, and products that are building the future of video. We are focused on artificial intelligence and other XR mediums such as augmented reality. Below you will find information regarding both traditional video tools and innovations and web3 specific products. Hopefully this research will help others experiment with video and also provide some insight into specific tools across various creator segments. We also briefly touched upon some of the challenges currently facing AI and touched on some ideas for how this technology can ethically and fairly monetise and credit artists. All research was conducted thanks to a grant from Livepeer, which provides decentralised video transcoding for web3 applications. We aim to use this research as we build out a Creator Development Kit for professional creators, which will live at our current XR hub - Art Official Intelligence. You can learn more about Livepeer here.
RESEARCH ABSTRACT
AI & Web3 Video: Tools, Trends & Resources for the Creator Economy is a research project aimed to dive into the past, present, and future of video. It’s estimated that 3.5 billion internet users will watch some form of online video content in 2023. Increased public consumption of video will put even more demand on videos to be high quality, innovative, and with no to minimal latency.
While mobile phones democratised access and rapidly shifted the most common viewing mode from horizontal to vertical), they didn’t alleviate many of the hurdles facing professional creators when it comes to creating high-quality video productions for their audience. The arrival of artificial intelligence in video will help to expedite and automate certain areas of the video creation and post-production process, creating new workflows and (hopefully) new long-lasting solutions.
Web3 video and technical solutions like Livepeer help video creators alleviate some of the costs and limitations when it comes to video uploading and processing. Current use cases such as token-gated access to live-streaming and video-on-demand content have the potential to be beneficial to the growth of the creator economy. However, it’s disruptive XR technologies like AI and AR that will once again redefine how we create, view, and interact with video. As web3 aims to build the next evolution of the creator economy, it’s important to be aware that the needs and pain points of creators vary across different mediums. The real innovation is likely to happen through the creation of niche solutions addressing unique challenges of various creative verticals.
Our research aims to highlight some of the video products being built in the web3 ecosystem, many which rely on the Livepeer tech stack. It also serves as a guide filled with tools and resources for professional creators looking to experiment and elevate their current content and productions. It will be interesting to see to what extent professionals adopt these new technologies, before they make it into the hands of the end user. It’s also clear that the intellectual property debate is far from over when it comes to who gets to create with what. Our guide will be a living document, updated with suggestions from the wider web3 community as we continue to ideate, build, and execute new solutions that leverage the powerful combination of the Livepeer tech stack, cryptocurrency, and blockchain technology.
A NOTE ON LIVEPEER 101
Livepeer is a pioneer in decentralised video infrastructure. Their mission is to power “video streaming applications at a highly efficient price, and infinite scale.” Making transcoding affordable has helped Livepeer disrupt live streaming, and much of their work also applies to video-on-demand and web3 video. Founded in 2017, Livepeer helps many of the challenges that come with uploading and processing video on the world wide web through a decentralised network comprised of orchestrators and delegators.
Dive into Livepeer with this 10-minute primer.
Learn more about building web3 video applications with the Web3 Builder Guides
Unlock long-form video via the Livepeer 10 GB Long Take NFT Publisher.
View Livepeer Token on Arbitrum
INTRODUCTION*
A brief history of video…
The history of video can be traced back to the late 1800s, with the invention of the first motion picture cameras. In the early days of film, movies were primarily shot on large and heavy cameras, and the film was distributed to theatres for viewing. It wasn't until the 1930s that home movie cameras and projectors became available, allowing people to view their own films in the comfort of their homes.
During the 1950s, the development of new technologies such as the VHS cassette and the camcorder revolutionised the way people watched and recorded video. The VHS cassette allowed people to record and play back television shows and movies, while the camcorder allowed people to easily record their own videos. With the advent of the internet and digital technology, video has become even more accessible with streaming platforms like YouTube, Netflix and many more making it easy to instantly view a vast library of content. Mobile phones have also made capturing video incredibly convenient. Anyone is able to shoot, edit, and upload content, with in-app tools available in many leading social media apps (ie: Instagram Reels, and Tik Tok video) . This has led to a new group of content creators shooting and editing video. Nowadays, video content has become a norm in our daily lives, it's used for entertainment, education, and even professional purposes.
Disruption across several art forms…
Video has disrupted several art forms. One of the most notable disruptions has been in the field of cinema. With the advent of video technology, it became much easier and cheaper to produce and distribute movies and television shows. The rise of streaming has led to an explosion of new content and led to a decline in the traditional Hollywood studio system. This has led to more diverse and independent voices in the film industry.
Additionally, video has also disrupted the field of visual art, as video installations and video art have become increasingly popular ever since their inception in the 1960’s. This medium allows for more dynamic and interactive forms of expression, and has also led to the creation of new art forms such as virtual and augmented reality art.
Video has also had an impact on the music industry, with music videos becoming an important medium for promoting and distributing music. This has led to the creation of a new form of art, the music video, and has also helped to establish the careers of many musicians and bands. The first promotional music video is believed to be Queen’s “Bohemian Rhapsody” (1975). However MTV brought music videos to the masses, with the first video premiering on the channel in 1981 aptly titled “The Video Killed the Radio Star.”
In general, video has allowed for new forms of expression and it has also made it easier for people to access and consume art. In 2021, the average person watched 100 minutes of video per day.
Current trends in video…
Live streaming: With the rise of platforms such as Twitch, YouTube Live, and Facebook Live, Instagram, and TikTok, live streaming has become increasingly popular. Almost every social platform includes a live streaming component (although Snapchat’s is not available to the public): This allows for real-time engagement between creators and viewers, and has led to a rise in gaming, e-sports, and other interactive content. In 2021, people consumed approximately 548 billion hours of streaming via mobile devices.
Short-form video: Platforms like TikTok and Instagram have popularised short-form video content, with videos typically being less than a minute long. This has led to the rise of the "creator economy" and influencer marketing.
Vertical video: With the increasing use of mobile devices to watch videos, vertical video (also known as Portrait Mode) has become more popular as it better fits the screen size of smartphones.
Virtual and augmented reality: With advancements in technology, virtual and augmented reality are becoming more accessible, and are increasingly being used in video content such as gaming, advertising, and entertainment.
Social causes: As people increasingly use video platforms to express themselves and to share their opinions, more and more people are using video to promote and raise awareness of social causes.
Personalisation: With the advancement of technology and data, video platforms are personalising content for each user, tailoring the content to their specific interests and preferences.
Animation and motion graphics: Animated and motion graphics videos are becoming more popular for explaining complex ideas, marketing purposes, and for creating engaging and interactive content. 91% of businesses are expected to use video as a marketing tool in 2023.
Interactive video: Interactive video allows viewers to interact with the content, making it more engaging and personalised. This technology is being used in various ways such as in e-learning, gaming, and advertising.
How are video-on-demand and live streaming used differently across professional creator verticals (music, fashion, gaming, art, content creation, film, food, design, etc)?
Video-on-demand and live streaming are used differently across professional creator verticals, depending on the specific needs and characteristics of each industry.
Music: Both video-on-demand and live streaming are used to promote and distribute music. Music videos are often used as a form of video-on-demand content, while live streaming concerts and music festivals have become increasingly popular.
Fashion: Video-on-demand is often used to promote fashion collections and runway shows, while live streaming is used to provide behind-the-scenes access to fashion events and to give real-time coverage of fashion shows.
Gaming: Live streaming is particularly popular in the gaming industry, as it allows players to interact with viewers in real-time and to showcase their skills. Video-on-demand is also used to provide highlights and replays of gaming events.
Art: Video-on-demand is often used to promote and document art exhibitions and installations, while live streaming is used to provide interactive tours and behind-the-scenes access to art events.
Content Creation: Both video-on-demand and live streaming are used by content creators to create, promote and distribute their content. Video-on-demand is used for pre-recorded shows, tutorials and vlogs, while live streaming allows for real-time interaction with the audience and to showcase their talents.
Film: Video-on-demand is often used to distribute and promote films and television shows, while live streaming is used to provide behind-the-scenes access to film productions and to showcase premiere events.
Food: Video-on-demand is often used to promote and document cooking and recipe tutorials, while live streaming is used to provide real-time cooking demonstrations and to showcase behind-the-scenes access to restaurants.
Design: Video-on-demand is used to showcase and promote design projects, while live streaming is used to provide interactive tours and behind-the-scenes access to design events.
Overall, while video-on-demand and live streaming are both used to promote and distribute content, the specific use cases and how they are implemented will vary depending on the industry and the needs of the creators and audiences.
* this introduction was created in collaboration with ChatGPT
Part One:
Examples of AR, AI, XR, and across various creator verticals
Film:
The film industry is ripe for disruption yet again, thanks to AI and the new storytelling capabilities that are unlocked with AR. Artists such as Sutu Eats Flies are already creating visual storyboards for films that don’t exist. Others are mashing together director aesthetics and classic films to create new visual worlds. Many entertainment outlets are hinting at a future where films are made completely with AI. And while the industry is moving at a rapid pace, the current results are fun, but primitive.
Film director, VFX specialist and animator Mick Mahler recently tried to make a CG short film using AI. While he was able to complete the project, a lot of post-production, editing software, and additional skills were required to achieve the final result. He also used non-AI tools such as Plot Generator to assist with specific parts of the pre-production process.
There are however quite a few tools emerging to help with specific areas of the multi-step film process.
Music:
Gorillaz have always been at the forefront of technology. The band was able to create an augmented reality concert utilising the ARCore Geospatial API from google. The goal of the global
Concert series was to “transform public spaces with cultural experiences.” Those wishing to experience the AR experience are required to download a mobile app. While AR technology continues to improve, the best user experience still requires the use of a mobile application.
Art:
Claire Silver collaborates with AI to explore themes of vulnerability, trauma, disability, social hierarchy, innocence, and divinity, and question the role they will play in our transhumanist future.”Her digital works are sold as NFTs and while many artists have been vocal about their anti-AI stance, Silver continues to push boundaries in her own artistic practice, while showcasing techniques and possibilities to her audience. The artist has also started to explore animation, providing tutorials on how to create moving visuals using the latent spacing technique. You can view the tutorial series here:
Fashion:
While Model Avatars have been around for a few years, there are a few challenges that stand in the way of AI innovation in the fashion industry. One of these is regarding 3D Modelling. While textures are being created using text2image generators, 3D modelling for the moment remains a tedious and time consuming skill. One tool that attempts to reduce the challenges of 3D Modelling is PIFuHD - a 3D modelling tool - view on Github.
AR on the other hand is poised to redefine the shopping experience. Snapchat and DressX have been experimenting with this for a while. And AR development platforms such as 8th Wall make it easy to develop interactive AR. While Time Magazine has stated that “augmented reality is the future of the shopping experience,” we’ve yet to see a mainstream adoption or desire for this by consumers.
Content Creation:
The rise of short-form video has led to a new category in the creator economy, the content creator. Because these creators are focused on bite-sized content, their use of AI tools is likely to be less predictable than in more traditional creator verticals. We can also expect content creators to take bigger risks with this technology, highlighting unexpected use cases and initiating trends that have the potential to be adopted by a wide sector of the population. A great example of how content creators are able to create global trends is the use of the AI Anime Filter on TikTok (as of January 15 2023, the #rareai had more than 387 million views on the app). TikTok has also launched Effect House, a beta AR tool which “makes it easy create, publish, and share high-quality augmented effects for TikTok.” Another dynamic use of AI technology for content creation is the Deepfake Tom Cruise which went from TikTok parody for full on AI company.
Types of AI Video Techniques:
Latent Space Animation: a model is trained on a dataset of real images to generate new, synthetic images. These images are typically generated by interpolating the latent space of the model, which is the underlying feature space that the model uses to represent the images it has been trained on.
2. GAN-based video: video that is generated using a Generative Adversarial Network (GAN), a type of deep learning model that is trained to generate new images that are similar to real images.
3. Style transfer video: video that is generated by applying the style of one image to the content of another image.
4. Deepfake video: video that is generated using AI to replace one face with another in a video.
5. Synthetic video: video that is generated using AI to create new video from scratch, without using any real footage.
6. Video synthesis video: video that is generated by synthesising new video frames based on a set of input images or video, using models such as VAEs (Variational Autoencoders) or Flow-based models.
7. 3D-generated video: video that is generated using AI to create 3D models and animations.
8. Augmented reality video: video that is generated by superimposing computer-generated images onto real-world footage, creating an augmented reality experience.
Part Two:
AI Video Tools & Resources
Here are a few of the top AI video tools, resources, and examples for content creators:
Metaphysic AI - AI content generation tools and infrastructure that lets users own and control their biometric data (home of the deepfake Tom Cruise)
Mimic AI - AI Photo Face Animator App
Prompt Muse - AI Character creation tutorials
Yepic AI - Multilingual AI Video Toolkit
Animated Mario - made with Midjourney and D-ID
Existing Tools for AI text2video creation
D-ID - Professional VideosGenerative AI tool to create customised videos featuring talking avatars at a touch of a button, for businesses and creator. Supercharged with Stable Diffusion and GPT-3
Synthesia - Create realistic AI videos in a matter of minutes with an AI presenter
Synths - Convert blog posts into a video with Human AI avatar and voiceover on autopilot
Elai - Create AI Videos from text
Gila Studio -Create videos from text and also utilise Avatar presenters
Image Animation & Latent Space Video Tools
Video Editing Tools
VEED - AI Video Editor
Lumen5- Uses AI to convert text to animated slides
Designs.AI- create design assets
Wisecut - AI Video Editor
Rawshorts - Explainer Video made with AI
Steve.AI - simple videos make with AI
Kapwing - Smart editing tools
Deepen AI - AI Powered Video annotation
Wibbitz - Online and AI Video editor
Tools in Research Phase:
Imagen Video - part of the Imagen text2anything project from Google
Phenaki - Long-form and text2video from Google
Part Three:
AI Video Use Cases
How can AI-powered content and Avatars may soon be used across the creator economy, helping to provide new sources of capital?
Fashion: Virtual fashion shows and clothes lines using AI models. This can help more members of the industry access capital and provide opportunities to work on an international scale while reducing carbon footprint due to travel. Issues such as fittings are also resolved as AI models will have fixed measurements.
Content Creators /Influencers: AI avatars as social media influencers can promote products and services in collaborations with brands. Paid livestream campaigns, unpacking videos, travel destinations and lifestyle content can all be showcased by AI generated figures.
In the future lifestyle influencers may use AI backgrounds and prompts to livestream from AI generated locations and make it appear that they are in “Wonderland.” We are already seeing this potential especially when images are generated using Midjourney.
Cooking TV shows: Recipe videos and cooking tutorials with AI hosts or avatars. While live streaming, creators could include a call to action button that allows viewers to fund their projects or buy a certain product.
Real estate: Create virtual home tours and property videos with AI hosts or avatars, or using depth2image to change the interior design while virtually visiting the property, using design prompts.
Coaching: Use AI avatars as virtual coaches or tutors in various fields like language, fitness, personal development
Sports: video highlights/analysis of sports games using AI avatars
Medicine: Virtual patient simulations and medical training videos made with AI avatars. >
Marketing: AI videos for adds or product demonstrations. This is already being explored by
Part Four:
Web3: Apps & Onboarding
State of the market for apps and Dapps incorporating web3 video:
It’s still early for web3 video, and very early for web3 video incorporating XR technology. While there are many projects in development we are focused on projects that have announced launch or are currently in beta, and which offer a pleasant UI/UX experience. This list is likely to grow over the next months:
Beem - Web3 video, live streaming and distribution interface. Beem offers advanced logic and token gating allowing for bespoke audience experiences.
Lenstube - A YouTube-esque platform which allows creators to monetise their audience through channel subscriptions, and feature called “collect.” Lenstube has had more than 10,000 videos created since its inception. A combination of short-form videos, known as Bytes and long-form videos exist on the platform.
The402 - Crypto-enabled livestreaming which allows for clip sharing (10, 30, 60 seconds)
Lensplay - Soon to be released mobile-first video sharing application on Lens Protocol
Rewind - Billed as the future of television from the creators of Blockbuster DAO - launching mid-January 2023
Kino - Crowdfunding Web3 film initiative
What on-ramps exist for professional creators between web2 and web3?
Currently one of the biggest challenges to creators entering Web3 are on-ramps. Most creators enter the space through DAOs, Websites, or Residency Programs. The growth team at Lens Protocol is working to onboard creators into the Lens ecosystem, which helps to provide new monetisation and community building opportunities. Current projects that are successfully bridging the gap between Web2 and Web3 with a creator focus:
What kind of education and systems need to be set up for onboarding creators (workflow, payments, tech101)?
Tech education: Resources such as tutorials, video guides, and documentation that can help them understand the basics of Web3, decentralised video, decentralised social graphs, and explainers regarding AI tool usage.
Payment educations: Crypto 101, How to use bridges, tax implications, and royalties are all important for new Web3 creators to understand. This also includes explaining how payment and royalties function for Web3 creators. There also need to be systems that allow split royalties, especially as different verticals work together for a single product.
Data sets: Creators access to high-quality "data sets" that they can use to train their AI models and generate new content without using other artists style like image2image. While in the initial experimental phase using commercial IP (Disney, Public Figures, etc) can be fun, it may lead to legal issues that can be avoided with more ethical data sourcing.
Community support: Access to a community of developers, creators, and experts that can provide support, mentorship, and feedback is necessary for the growth of the creator economy. The Metaverse could be used as an immersive educational tool to make a for a more impactful experience.
Webinars: Online workshops and other events that cover different topics like AI content creation use cases, monetisation, decentralised storage, etc. These exist but may need to also be shared with people outside of the Web3 community to help democratise access.
Onboarding on demand: 1-on-1 support is still needed to ensure comprehension and completion of steps needed to participate in Web3 and emerging tech, including setting up of accounts, security, and education around how to use different tools and platforms.
Success stories: Spotlights on success stories and innovative use cases can help to drive interest and adoption.
Part Five:
Augmented Reality
What are the current AR solutions that video creators and developers can utilise to build new applications?
Currently mobile-first AR experiences are far beyond what’s capable in a powered web application (PWA). While PWA solutions will be ideal for wider adoption, AR technology works best in an application environment. These are some of the top AR solutions and research labs
Part Six:
Additional Research
How can you solve royalties in AI usage of existing IP?
The ethics around Artificial Intelligence will be a spotlight issues in 2023. It’s difficult to say how this will end, however it’s clear that both the tech community and artists community have differing opinions on the matter. While the process of AI generation may not directly copy specific artist styles, the datasets used to train some of these models may include information and portfolios that were obtained in questionable ways. Spawning AI, which is focused on helping artists have ownership of their training data created the website, Have I Been Trained. The tool allows artists to see if their work has been used and opt-out of its inclusion in the Stable Diffusion V3 model.
A new lawsuit has been filed against the company
https://stablediffusionlitigation.com/
, which joins the Nov 2022 lawsuit which was filed against Github Co-Pilot, which uses AI to generate computer code. These ruling could shape the future of AI for commercial use, although the U.S. Copyright Office already ruled that AI art can’t be copyrighted.
What are the most important blocks to be built for the creator economy?
Monetisation is already available for creators via Lens Protocol and accessible tools such as the Long Take NFT publisher. Smart contracts can be deployed easily via Nifty Kit and Manifold. However education is vital to adoption, innovation, and usage. Some of the pieces that are key to bringing in the next 1 million creators include:
Friendly user interfaces where you can easily create, view on demand or livestream AI generated content
Community assistance - Accessible ways to learn and receive support. Discord has been a standard, but applications such as Common Ground are emerging to improve the user experience.
Tutorials and educational videos that utilise emerging tech and features such as AI Avatars to give context and examples of how these XR tools work
Community building through both IRL and URL Events
Bespoke products that solve problems within individual creator segments. An example of a niche product is Lyrical.world which helps musicians create visual worlds for their songs using artificial intelligence.
Some ideas for specific tools that can help creators excel in this new frontier include:’
Integration with existing Web2 tools and platforms (such as Instagram):
A robust suite of AI Solutions including:
Text2text
Text2image
Image2image
Text2avatar
Training custom AI models
AI video tools:
-Videos with avatar presenters
-Blog/pdf/URL creation with text/audio to video/animation solutions
-Editing tools such as remove background/objects/noise
-Accessibility through subtitles and multiple languages
-Video2text for transcriptions and meeting notes
- Text2slides/animated slices
How does Livepeer power video across different apps in the web3 social ecosystem?
While many projects are still in development, many builders are choosing to pair the Livepeer Tech Stack with the Lens social graph. The pairing allows creators to easily create, mint, and monetise content. Currently video is used across the Lens social graph by
Huddle - Token-gated meeting via dashboard. it can also be used to leverage their infrastructure to build real-time audio/video apps
Lenstube - Video sharing platform similar to YouTube. It also includes short-form video known as Bytes
Lensport- A marketplace for creators
Lenster- Social media app that supports video and audio posts
The 402 - Token-gated live streaming
Phaver - Aggregator which curates interesting content across the ecosystem
This list is expected to grow throughout 2023.
Other blockchains and web3 social protocols that have integrated the Livepeer tech stack:
What is the state of the market for Apps /Dapps aimed at tools for the creator economy?
Generally speaking I believe we are currently experiencing a RAPID growth and expansion. There is a high supply of products and demand is yet to be tested). As new and advanced AI solutions emerge the ones likely to succeed are those that enable creators to generate, edit, and monetise their content in innovative and faster ways. Some of the most popular solutions include text2text, text2image, image2image, text2video (animated slides, short videos not generated with AI).
The current supply growth is driven by the rapid evolution of open source AI algorithms like Stable Diffusion, where a huge community of builders has decided to create their own platforms and tools using these existing APIs. These solutions are cost effective and customisable, which means new ones will continue to appear. Due to blockchain tech, creators have more power and control over their creations, allowing them to monetise and distribute their content in a more autonomous and efficient way. Will there be space for all of these products, especially once Google and Meta exit the research phase?
Conclusion:
We live in an exciting time where technology has become intertwined with our everyday life. The combination of XR and Blockchain technologies will allow creators to innovate a rapid speed while also finding new avenues for sovereignty.
And while video is expected to play a large part in Web3, the use cases and applications are still primitive. Many platforms are in the exploratory phase, and while products continue to be built there is likely to be a large supply that might not match demand for some time.
Ideas will remain the magic sauce and humans that are able to leverage the capabilities of AI and XR technology to execute their visions are poised for the most success. While many fear that XR technology will take their jobs, like every revolution in history, new jobs and opportunities will appear even while others may become obsolete.
Additional Resources:
State of AI 2022 from McKinsey - Top 10 Takeaways from @thealexbanks
Future Tools: Directory of 400+ AI tools broken down by category
Replica Studios- Text to Speech
Omniverse -AI powered facial animation from Nvidia
Music AI Ecosystem from Water and Music
AI Films AI - Billed as the new Netflix
Layer.AI - Game asset creation platform
AI Breakfast on Twitter - AI resource’
Selas AI - French-based AI generation studio
Mixamo- 3D character animation
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation