Day 3-Key Synthetic Image Data Generation Tools 

By the end of today's lesson you should :

• know enough about each tool and workflow to decide which one to use in building your portfolio.

These are the lesson objectives. In regards to synthetic image data generation :

• Learn how and when to use Blender 

• Learn how and when to use Unity Engine 

• Learn how and when to use Unreal Engine 

• Learn how and when to use Nvidia Isaac Sim.

Let's begin then.

BLENDER 

Blender was my very first introduction to synthetic image data generation, and I think that’s why I suggest most people start with it.  

I recommend beginners use Blender because I believe it's easier to set up and run your first generation.  

If you’re already a Blender user, then it’s going to be even easier for you.  

If you're not familiar with the Python programming language, fear not.  

The instructor (Kevin Kelly) recommended below teaches in a very beginner-friendly approach. I wasn’t very familiar with Python programming, but I was able to follow him, generate a successful dataset, and even complete a small computer vision project alongside.

When to use Blender?

I recommend this Blender path if the following apply to you:

- You want to create your first synthetic image dataset super fast.

- You don't have a powerful laptop.

- You are a complete beginner to the world of synthetic image data generation.

- You aren’t familiar with Python.

Also, most of the assets you would use in more advanced software will likely need to be created in Blender, so it’s a good starting point.

WHEN TO USE UNITY ENGINE

Here, I’m specifically referring to the Unity Perception Package 1.0.  : 

Even though it's no longer being updated by the Unity Perception team, I think it's a great next step after Blender.  

Here’s why:

- The setup is really fast.  

- There is detailed documentation on how to set up and run your first generation. Below, I’ll provide all the links and the sequence in which I think you should study the learning materials.  

- It’s not resource-intensive in terms of computer demands. With a decent gaming machine, you can create some amazing synthetic image datasets.

WHEN TO USE UNREAL ENGINE 

Now, I want to start by sharing that this approach is very new to me, and I am still exploring it at the time of writing this lesson. Therefore, I can only recommend a general approach to using Unreal Engine for synthetic image data generation. There isn’t a lot of beginner-friendly learning material on the topic, so you’ll need to do a lot of self-exploration and experimentation to get what you need.

If you already use Unreal Engine for much of your technical artwork, I highly recommend skipping Unity Perception exercises and focusing on learning how to use Unreal Engine to achieve the desired result.

But how can you know the desired result if you’re unsure what’s expected for synthetic image datasets?

I suggest the following steps:

1. Review the learning materials I recommended in Day 1’s Daily Challenge (scroll to the bottom of Day 1 to find it) to get familiar with industry expectations.

2. Watch the Unity Perception tutorial (https://youtu.be/mkVE2Yhe454?si=whRRh5TXTGcNLX1_)  to get an idea of how the engine generates these datasets.

3. Open the Unity Perception code to understand how it works and consider how to create similar logic using Blueprints or C++ in Unreal Engine. The code is in C# and is usually located at:  "C:\Users\[YOUR PC'S NAME]\Documents\Unity Projects\cnametest\Packages\com.unity.perception@1.0.0-preview.1\Runtime\GroundTruth\Consumers\PerceptionEndpoint"

WHEN TO USE NVIDIA ISAAC SIM

Another disclaimer: I have only used the NVIDIA Omniverse Code Replicator. However, about three months ago, I received this comment on my channel (see Fig. 1.0).

I believe the Replicator extension is no longer supported in the NVIDIA Omniverse family of tools, as their focus now seems to be on making it available only in Isaac Sim. I haven’t downloaded Isaac Sim yet because it’s a large file (around 50 GB), but I plan to do so at a later time. Here’s when I think any technical artist interested in synthetic image data generation might consider moving from Blender to NVIDIA Isaac Sim:

1. You have a laptop powered by a GTX or RTX 3070 or higher. You can check the official website for the minimum system recommendations here: https://docs.omniverse.nvidia.com/isaacsim/latest/installation/requirements.html

2. You’re comfortable with Python, as it’s the programming language used to generate datasets.

Now, here’s my recommended learning path for each tool. Generally, we can identify three primary learning paths. Of course, there are more, but for the sake of speed, let’s focus on these three.

For Unreal I haven’t spent much time on them but I think once you are a user of Unreal Engine, you should be abe to figure out how to use it for synthetic image data generation with the guide above. I am also sharing resources below to help your learning be a lot smoother:

1.Why you need Quixel add-on for synthetic image generation in Unreal Engine during Installation

Why?

You would get a lof textures, objects for free.

This would help you create more realistic scenes and move a whole lot faster.

Link:https://youtu.be/wd_sdFaYdIk?si=i66rAc-k2Q9AkNgF

2. What is possible with Unreal Engine 5 when it comes to Synthetic Image Generation

Why?

This is just to open your mind to possibilities you probably didn't even think of, and honestly to just get you excited.

Link: https://youtu.be/t5dgMfF8xSE?si=bIrgLcda3XEFFAgM

3. Beginner-friendly game development tutorial

Why?
There are far more tutorials out there on game development in Unreal Engine than Synthetic Image Generation.

I figured if I want to get familiar with the interface and some of the tools I would use it's better to build along with a game tutorial first.

Link: https://youtu.be/1XjgLKrb4_M?si=J2rKbjkeSUOfpabT

4. Create 2D bounding boxes in Unreal Engine 5 for Synthetic Image Generation

Why?

You need 2D Bounding Boxes for all objects, people and other enviromental elements you generate.

Link:https://youtu.be/zaPAlBaMZE4?si=IXVsen4pvFeBO_Fp

5. Create a Segmentation Mask in Unreal Engine

Link: https://medium.com/@jeff_97181/generating-image-segmentation-datasets-with-unreal-engine-4-2b5b9f75da34

6. Create Segmentation Mask in Unreal Engine/Create binary segmentation images based  

Link :https://forums.unrealengine.com/t/create-binary-segmentation-images-based-on-object-flags/397584/2?u=inkman_eli

7. Unreal Python API for Synthetic Image Generation

Why?

Python offers some capablities you might need on your journey.
My focus is to use more Blueprint Visual Scripting but it's good to know this a bit

Link:https://youtu.be/Pxz8PAQR6_g?si=KVbfosN145W-sbHn

8. Creating Digital Twins in Unreal Engine 

Why?

Just helping you see the possiblites and getting you excited.

Link:https://youtu.be/Hrv-T-sWL7w?si=N3SgaSqWzekzRvEu

9.Share your Blueprints and Learn from others

Why?

Taking screenshots of your blueprints can be a challenge sometimes when the setup is too long.

This free tool helps you solve that problem and also learn how others setup their blueprints.
Link to Demo video :https://youtu.be/DH6cR9b3m04?si=Kxcc2i0E-TR86Lc-

Link to tool: www.blueprintue.com

10. Fix Unreal Engine Blueprint Editor Lagging:
Why?
By default Unreal Engine places it's settings to the highest and this can slow down the software.

Make sure to restart your  sotware when you are done.

Link: https://www.reddit.com/r/unrealengine/comments/11uf2q2/ue5_blueprint_editor_lag_

11. Fix Unreal Engine Lagging

Why?
Slow softwares are unpleasant to work with. 

Link:  https://youtu.be/esrnQBq75qg?si=U5V3clRZ4LQ9zQUp

12. Fix Lighting Needs to Be Rebuilt in Unreal Engine

Why?

Calculating light in Unreal Engine can be tasking so sometimes it doesn't load properly, I get this all the time in Unreal Einge 4.27 but not 5.2.

This is here just in case it pops in any 5+ versions .

Link:https://youtu.be/blwjnwvLgq8?si=9TD6H4LFE7bvCYPO

13. Speed up your frame rate in Unreal Engine

Why?

Faster FrameS Rate Per Seconds gives for a better working experienc.

Link:https://youtu.be/z_gPaqa_63Q?si=SRFZORqcTkK6Edtx

14. Fix Unreal Engine FrameS Per Seconds (FPS) stuck at 60 FPS

Link:https://forums.unrealengine.com/t/stuck-at-60-fps-with-smooth-rate-disabled/398803/6?u=inkman_eli

Tip:

If you can't find the Nvidia Experience Settings don't worry, just complete the second half of the solution in Unreal Engine, restart the software and it should work.

15. Fix Unreal Engine FrameS Per Seconds (FPS) stuck at 60 FPS

Link:https://forums.unrealengine.com/t/stuck-at-60-fps-with-smooth-rate-disabled/398803/6?u=inkman_eli

Tip:

If you can't find the Nvidia Experience Settings don't worry, just complete the second half of the solution in Unreal Engine, restart the software and it should work.

I also haven’t spent much time with Isaac Sim but I think following the documentation should help you get up and running. I would have recommended the Nvidia Omniverse Code Replicator but it’s been discontinued. I still have the old versions but they can't be downloaded by new users from the Omniverse Launcher App.

Link to documentation for Isaac Sim: https://docs.omniverse.nvidia.com/isaacsim/latest/index.html

-So that means Day 3 of the series would become Day 1 of the learning path listed above, then Day 4 of the series becomes Day 2 of the learning plan and so on.

-What I would do for the next 45 days or so of study is to list:

• A summary of what you would learn on each day

• Areas that you might find confusing and tips to bring more clarity

• Practice exercises you could do to reinforce what you just studied.

The goal for Day 3-Day 45  of this  series is to grasp a firm understanding of the basics of the  synthetic image data generation tools and workflow through practice so that you can begin working on your portfolio pieces from Day 22 onward. Then we start sending your work to the companies we found along with job applications.

So for Day 3 we would begin creating your first synthetic image dataset in Blender.

Challenge for the Day

Practice: Immediately begin watching Day 3’s lesson= https://youtu.be/E1Pqpfg5kSo?si=h87Sa9RoVDOhvJLz

I would recommend to first watch the entire video without building along, then rewatch it and build along. I personally find that to be a more effective way to retain practical skills from online video lessons.

Until the next one, this is Eli-Stay exceptional.