Digital twin-enabled Smart Shipping Workstation w/ Omniverse

November 25, 2024

Navigation Bar

Apply Theme

Exploring the digital twin synthetic data generation and AI-oriented advancements on real-world shipping operations w/ NVIDIA Omniverse.

Keywords
Brands
Hardware
  • 1ELECROW Custom PCB
  • 1Raspberry Pi 5
  • 1Arduino Nano Matter
  • 1USB Webcam (PK-910H)
  • 4Nema 17 (17HS3401) Stepper Motor
  • 4A4988 Driver Module
  • 1Tiny (Embedded) Thermal Printer
  • 1SSD1306 OLED Display (128x64)
  • 1Infrared (IR) Break-beam Sensor (300 mm)
  • 2Micro Switch with Pulley (KW10-Z5P)
  • 1Logic Level Converter (Bi-Directional)
  • 3Button (6x6)
  • 15 mm Common Anode RGB LED
  • 1ATX Power Supply Unit (PSU)
  • 1XH-M229 ATX Power Supply Adapter Board (Breakout)
  • 3DC Barrel to Wire Jack (Male)
  • 2DC Barrel Female Power Jack
  • 15 mm Steel Balls (Beads) for Bearings
  • 1M3 Screws
  • 1M3 Brass Threaded Inserts
  • 1Jumper Wires

Description

At the pinnacle of industrial artificial intelligence and machine learning applications, a digital twin represents a virtual construction of real-world physical products, mechanisms, or mechanical procedures. Since the simulation of a real-world product or industrial technique provides the flexibility of exerting countless examination scenarios, even to the point of cycling arduous or dangerous tasks, without causing any tangible ramifications and safety risks, digital twins hold immeasurable importance in developing adaptive product manufacturing procedures and building secure, cost-effective, and efficient industrial facilities.

After inspecting recent research papers about the applications of digital twins in industrial operations, I noticed the focal point of employing a virtual representation is to improve the safety and efficiency of an already existing industrial facility or mechanical process. Even though forestalling acute physical hazards due to dangerous work safety risks and advancing the precision of ongoing industrial operations are the prominent digital twin use cases, I wanted to explore the innovative opportunities of reversing the digital twin implementation and starting with a virtual industrial construction, consisting of individual machinery and sample product components, to develop a safe, practical, cost-effective, and efficient real-world mechanism from scratch.

By reversing the digital twin application process, I wanted to investigate whether having a virtual construction before building the real-world counterpart could help to forfend concomitant risks of assembling an industrial manufacturing system, reduce exorbitant overhaul costs due to the lack of internal design blueprints, and test device components to obtain the optimum performance for multifaceted operations.

As I was conceptualizing this proof-of-concept project, I inspected various industrial settings with which I could show the benefits of reversing the digital twin implementation. Since product transportation and shipping operations require complex industrial mechanisms to achieve accuracy and reliability while maintaining a time-sensitive workflow, I decided to apply my reverse digital twin approach to design a virtual shipping workstation, construct a synthetic data set of customized sample products, and train a precise object detection model to accomplish building a production-ready product transportation mechanism. In accordance with my approach, I designed all sample products from scratch to emphasize the strengths of a full-fledged digital twin, providing the opportunity to train an object detection model for products waiting to be manufactured.

Since I needed to know the exact electronic components employed by the shipping workstation to create compatible 3D parts, I decided to prototype the mechanical device structure and design a unique PCB (inspired by Wall-E) based on Arduino Nano Matter as the workstation control panel. I designed the Wall-E PCB outline and encasement on Autodesk Fusion 360 to place the electronic components in relation to the PCB easily while designing the virtual shipping workstation.

After testing electronic components and completing the PCB layout, I designed a plethora of 3D parts on Autodesk Fusion 360, including but not limited to custom bearings optimized for 5 mm steel balls, planetary gear mechanisms, and separated rotating platforms. After finalizing the required mechanical 3D parts, I exported the virtual shipping workstation as a single file in the OBJ format to produce an accurate virtual representation of the shipping workstation.

Then, I imported the virtual shipping workstation into the NVIDIA Omniverse USD Composer, which allows users to assemble, light, simulate, and render large-scale scenes for world-building. To generate a realistic scenery for shipping operations, I utilized some free 3D models provided by Omniverse and designed some additional assets. After completing my shipping warehouse scenery, I experimented with camera, material, and lighting configurations to render the virtual shipping workstation with exceptional quality and produce a precise digital twin.

After employing built-in NVIDIA Omniverse features to construct my synthetic data set of customized sample products as instantiated by the shipping workstation digital twin, I uploaded the collected samples to Edge Impulse to train an advanced object detection model (FOMO) with synthetic product images. After validating and testing my FOMO model, I deployed it as a Linux (AARCH64) application (.eim) compatible with Raspberry Pi 5.

After building my object detection model successfully and completing my assignments with the shipping workstation digital twin on NVIDIA Omniverse, I started to print all workstation 3D parts to assemble the real-world counterpart.

To create a fully functioning smart shipping workstation with state-of-the-art features, I developed a web application from scratch to manage the MariaDB database server hosted by Raspberry Pi 5, run the Edge Impulse FOMO object detection model, and transfer the detection results with the modified model resulting images. I also developed a mobile application (Android) operating as the workstation interface and the proxy between the workstation control panel (based on Arduino Nano Matter) and the web application.

So, this is my project in a nutshell 😃

Please refer to the following tutorial to inspect in-depth feature, design, and code explanations.

🎁🎨 Huge thanks to ELECROW for sponsoring this project with their high-quality PCB manufacturing service.

🚀🤖 You can utilize ELECROW PCBA services to manufacture your designs, which provides $20 PCBA COUPONS for new accounts.

🎁🎨 Also, huge thanks to Anycubic for sponsoring an Anycubic Kobra 2 and an Anycubic Kobra 2 Max.

project_image_1
project_image_2
project_image_3
project_image_4
project_image_5
project_image_6
project_image_7
project_image_8
project_image_9
project_image_10
project_image_11
project_image_12
project_image_13
project_image_14
project_image_15
project_image_16
project_image_17
project_image_18
project_image_19
project_image_20
project_image_21
project_image_22
project_image_23
project_image_24
project_image_25
project_image_26
project_image_27
project_image_28
project_image_29
project_image_30
project_image_31
project_image_32
project_image_33
project_image_34
project_image_35
project_image_36
project_image_37
project_image_38
project_image_39
project_image_40
project_image_41
project_image_42
project_image_43
project_image_44
project_image_45
project_image_46
project_image_47
project_image_48
project_image_49
project_image_50
project_image_51
project_image_52
project_image_53
project_image_54
project_image_55
project_image_56
project_image_57
project_image_58
project_image_59
project_image_60
project_image_61
project_image_62
project_image_63
project_image_64
project_image_65
project_image_66
project_image_67
project_image_68
project_image_69
project_image_70
project_image_71
project_image_72
project_image_73
project_image_74
project_image_75
project_image_76
project_image_77

Design process, available features, and final results

As my projects became more intricate due to complex part designs, multiple development board integrations, and various features requiring interconnected networking, I decided to prepare more straightforward written tutorials with brevity and produce more comprehensive demonstration videos showcasing my entire design process, results, and device features from start to finish.

Thus, I highly recommend watching the project demonstration videos below to inspect my design process, the construction of the synthetic data set, and all of the shipping workstation features.



Step 0: A simplified illustration of interconnected networking

As a part of preparing a more visually-inclined tutorial, I decided to create a concise illustration of interconnected networking infrastructure to delineate the complicated data transfer procedures between different development boards, complementary web, and mobile applications.

project_image_78

Step 1: Testing electronic components and prototyping the device structure

Before proceeding with designing 3D parts, I needed to determine all electrical components required to operate the real-world shipping workstation. Thus, I started to test and prepare electronic components for prototyping the device structure.

#️⃣ Since Arduino Nano Matter is a versatile IoT development board providing state-of-the-art Matter® and Bluetooth® Low Energy (BLE) connectivity thanks to the MGM240SD22VNA wireless module from Silicon Labs, I decided to base the shipping workstation control panel on Nano Matter.

#️⃣ Since I envisioned a fully automated homing sequence for the moving workstation parts, I decided to utilize an IR break-beam sensor (300 mm) and two micro switches (KW10-Z5P).

#️⃣ By utilizing a soldering station for tricky wire connections, I prepared the mentioned components for prototyping.

project_image_79
project_image_80
project_image_81
project_image_82
project_image_83
project_image_84

#️⃣ Since I needed to supply a lot of current-demanding electronic components with different operating voltages, I decided to convert my old ATX power supply unit (PSU) to a simple bench power supply by utilizing an ATX adapter board (XH-M229) providing stable 3.3V, 5V, and 12V. For each power output of the adapter board, I soldered wires via the soldering station to attach a DC-barrel-to-wire-jack (male) in order to create a production-ready bench power supply.

project_image_85
project_image_86

#️⃣ Since Nano Matter operates at 3.3V and the IR break-beam sensor requires 5V logic level voltage to generate powerful enough signals for motion detection, the sensor cannot be connected directly to Nano Matter. Therefore, I utilized a bi-directional logic level converter to shift the voltage for the connections between the IR sensor and Nano Matter.

#️⃣ Since I planned to design intricate gear mechanisms to control the moving parts of the real-world shipping workstation, I decided to utilize four efficient and powerful Nema 17 (17HS3401) stepper motors, similar to most FDM 3D printers. To connect the Nema 17 stepper motors to Nano Matter securely, I employed four A4988 driver modules.

#️⃣ As a practical shipping workstation feature, I decided to connect a tiny (embedded) thermal printer to Nano Matter to print a shipping receipt for each completed order. I utilized a sticker paper roll to make receipts fastenable to cardboard boxes.

#️⃣ To build a feature-packed and interactive workstation control panel, I also connected an SSD1306 OLED display, three control buttons, and an RGB LED to Nano Matter.

#️⃣ As depicted below, I made all component connections according to available and compatible Arduino Nano Matter pins.

// Connections
// Arduino Nano Matter :
//                                Nema 17 (17HS3401) Stepper Motor w/ A4988 Driver Module [Motor 1]
// 3.3V    ------------------------ VDD
// GND     ------------------------ GND
// D2      ------------------------ DIR
// D3      ------------------------ STEP
//                                Nema 17 (17HS3401) Stepper Motor w/ A4988 Driver Module [Motor 2]
// 3.3V    ------------------------ VDD
// GND     ------------------------ GND
// D4      ------------------------ DIR
// D5      ------------------------ STEP
//                                Nema 17 (17HS3401) Stepper Motor w/ A4988 Driver Module [Motor 3]
// 3.3V    ------------------------ VDD
// GND     ------------------------ GND
// D6      ------------------------ DIR
// D7      ------------------------ STEP
//                                Nema 17 (17HS3401) Stepper Motor w/ A4988 Driver Module [Motor 4]
// 3.3V    ------------------------ VDD
// GND     ------------------------ GND
// D8      ------------------------ DIR
// D9      ------------------------ STEP
//                                Tiny (Embedded) Thermal Printer
// D0/TX1  ------------------------ RX
// D1/RX1  ------------------------ TX
// GND     ------------------------ GND
//                                SSD1306 OLED Display (128x64)
// A4/SDA  ------------------------ SDA
// A5/SCL  ------------------------ SCL
//                                Infrared (IR) Break-beam Sensor [Receiver]
// A6      ------------------------ Signal
//                                Control Button (A)
// A0      ------------------------ +
//                                Control Button (B)
// A1      ------------------------ +
//                                Control Button (C)
// A2      ------------------------ +
//                                Micro Switch with Pulley [First]
// A3      ------------------------ +
//                                Micro Switch with Pulley [Second]
// A7      ------------------------ +
//                                5mm Common Anode RGB LED
// D10     ------------------------ R
// D11     ------------------------ G
// D12     ------------------------ B
project_image_87
project_image_88
project_image_89
project_image_90
project_image_91
project_image_92

#️⃣ Furthermore, I put Raspberry Pi 5 into its aluminum case providing a cooling fan to secure all cable connections.

project_image_93

Step 1.1: Designing the Wall-E-inspired PCB layout and silkscreen graphics

As I was prototyping the device structure and conceptualizing the workstation features, I pondered the question of how I should design a unique PCB for a smart shipping workstation. Then, I remembered the perennial efforts of Wall-E to move and arrange garbage as small packages. Thus, I drew my inspiration from Wall-E while designing this PCB, running an automated package-moving operation :)

To simplify the PCB integration and place electronic components precisely while designing complementary 3D parts, I created the Wall-E PCB outline and a snug-fit PCB encasement on Autodesk Fusion 360.

project_image_94
project_image_95
project_image_96
project_image_97
project_image_98

Then, I imported my outline graphic to Kicad 8.0 in the DXF format and designed the Wall-E PCB layout and silkscreen graphics according to the prototype electronic component connections.

project_image_99
project_image_100
project_image_101
project_image_102
project_image_103
project_image_104

Step 1.2: Soldering and assembling the Wall-E PCB

After completing the Wall-E PCB design, I utilized ELECROW's high-quality PCB manufacturing services. For further inspection, I provided the fabrication files of this PCB below, or you can order it directly from my ELECROW community page.

#️⃣ After receiving my PCBs, I attached all electronic components by utilizing a TS100 soldering iron and the soldering station.

📌 Component assignments on the Wall-E PCB:

A1 (Headers for Arduino Nano Matter)

DR1, DR2, DR3, DR4 (Headers for A4988 Stepper Motor Driver)

Motor1, Motor2, Motor3, Motor4, (Headers for Nema 17 [17HS3401] Stepper Motor)

SSD1306 (Headers for SSD1306 OLED Display)

Thermal1 (Headers for Embedded Thermal Printer)

L1 (Headers for Bi-Directional Logic Level Converter)

IR1 (IR Break-beam Sensor [Receiver])

IR2 (IR Break-beam Sensor [Transmitter])

SW1, SW2 (Micro Switch [KW10-Z5P])

C1, C2, C3 (6x6 Pushbutton)

D1 (5 mm Common Anode RGB LED)

J_5V_1, J_12V_1 (DC Barrel Female Power Jack)

J_5V_2, J_12V_2 (Headers for Power Supply)

project_image_105
project_image_106
project_image_107
project_image_108
project_image_109
project_image_110
project_image_111
project_image_112
project_image_113

After concluding soldering components to the Wall-E PCB, I tested whether the PCB worked as expected or was susceptible to electrical issues; I did not encounter any problems.

project_image_114
project_image_115

Step 2: Creating a fully functional virtual shipping workstation

Even though I decided to build a virtual shipping workstation to present my reverse digital twin approach as a proof-of-concept project, I focused on designing an intricate mechanism manifesting an industrial-level shipping operation providing a professional product transportation system and a custom warehouse management system.

I designed all shipping workstation 3D parts on Autodesk Fusion 360, including custom mechanical parts for moving components.

While designing 3D parts, I utilized some third-party CAD files to obtain accurate measurements and create a precise virtual construction.

Nema 17 (17HS3401) Stepper Motor | Inspect

Raspberry Pi 5 | Inspect

In the following steps, I will explain my design process for each 3D part categorically.

After finalizing all 3D parts, I exported the virtual shipping workstation as a single file in the OBJ format to produce an accurate virtual representation of the shipping workstation.

project_image_116
project_image_117
project_image_118
project_image_119
project_image_120
project_image_121
project_image_122

In accordance with my reverse digital twin approach, I had to envision each virtual shipping workstation 3D part appearance as close as possible to their real-world counterparts since I needed to construct a synthetic data set and train an object detection model even before printing these 3D parts. Therefore, while designing, I decided on the filaments I would utilize for each 3D part. Then, I searched for material type and color code for each filament to assign them to the corresponding 3D parts on Fusion 360.

I selected these PLA filaments for different 3D parts.

🎨 For shipping workstation 3D parts:

  • ePLA-HS Grey (#B5B8BE)
  • ePLA-Matte Mint Green (#6DA582)
  • ePLA-Matte Peach Pink (#F9C0CF)
  • ePLA-Matte Deep Black (#2F3231)
  • ePLA-Matte Milky White (#F7F5F4)
  • ePLA-Matte Light Khaki (#AD9E8D)
  • ePLA-Matte Tangerine (#E24C13)
  • ePLA-Matte Almond Yellow (#C7D58C)

🎨 For sample product 3D parts:

  • eSilk-PLA Lime (#9CE40C)
  • eSilk-PLA Silver (#AAA3B5)
  • eSilk-PLA Jacinth (#FF8472)
  • ePLA-Metal Antique Brass (#CB9E70)
  • PLA+ Light Blue (#37B8F5)
  • PLA+ Fire Engine Red (#A62E34)
project_image_123
project_image_124

Step 2.1: Designing mechanical 3D components

Since I wanted to design the shipping workstation from the ground up, I decided to create custom mechanical components for the moving workstation parts.

#️⃣ First, I started to work on designing a template for bearings optimized for 5 mm steel balls. In this regard, I was able to create ball bearings in different sizes to swivel mechanical components.

#️⃣ For a simple assembly process, I designed the bearing template in three parts: inner ring, top outer ring, and bottom outer ring. The outer ring (top and bottom) includes M3 screw holes to adjust the bearing tightness easily.

#️⃣ Since I used parameters to define the dimensions and clearances of the bearing template, I was able to create custom bearings in different sizes effortlessly.

project_image_125
project_image_126
project_image_127
project_image_128
project_image_129
project_image_130
project_image_131
project_image_132

After completing the ball bearing template, I started to work on designing the two rotating platforms for storing and presenting sample products respectively.

Since I wanted to create an industrial-level shipping workstation and showcase the digital twin capabilities for intricate mechanical components, I decided to design planetary gear mechanisms to rotate the platforms.

To generate gears in different sizes, I utilized the SpurGear add-in script.

After experimenting with virtual planetary gear configurations, I decided to fix the ring gear and employ the planet carrier (Y-shaped) to attach and rotate the platform face. In this configuration, the sun gear behaves as the driver gear and provides higher torque while maintaining a lower speed.

To determine gear ratios and teeth numbers, I applied these equations:

🔢 Variables

  • R ➡ Ring gear teeth number
  • S ➡ Sun gear teeth number
  • P ➡ Planet gear teeth number
  • Tr ➡ Ring gear rotation
  • Ts ➡ Sun gear rotation
  • Ty ➡ Planet carrier (Y-shaped) rotation

🔢 Equations

  • R = (2 × P) + S
  • (R + S) × Ty = (R × Tr) + (Ts × S)

🔢 Since the ring gear is fixed and I wanted to have a 1/3 gear ratio for Ty/Ts:

  • (R + S) × Ty = Ts × S
  • Ty = Ts × (S / (R + S))
  • R = 96
  • S = 48
  • P = 24
project_image_133

#️⃣ According to the fixed ring planetary gear configuration, I designed the platforms with the embedded ring gear.

⚙️ First platform:

project_image_134
project_image_135
project_image_136
project_image_137

⚙️ Second platform:

project_image_138
project_image_139
project_image_140
project_image_141

#️⃣ Then, I designed the planet gears, the sun gear, and the secondary stepper motor gear. In this regard, the Nema 17 stepper motor attached to the platform drives the secondary gear to rotate the sun gear which drives the planet gears.

⚙️ First platform:

project_image_142
project_image_143
project_image_144
project_image_145
project_image_146

⚙️ Second platform:

project_image_147
project_image_148
project_image_149
project_image_150
project_image_151

#️⃣ After completing the planetary gear mechanisms, I designed the Y-shaped planet carrier connected to the planet gears via custom bearings.

#️⃣ To stabilize torque distribution, the sun gear and the planet carrier are connected to the central shaft of the platform via custom bearings.

#️⃣ Then, I designed platform faces attached to the Y-shaped planet carriers via snap-fit joints.

#️⃣ Since the first rotating platform stores sample products and the second rotating platform presents the transported product, I designed face separators and rotation pins accordingly.

#️⃣ The rotation pins are tailored for the selected platform homing methods — IR break-beam sensor and micro switch.

⚙️ First platform:

project_image_152
project_image_153
project_image_154
project_image_155
project_image_156
project_image_157
project_image_158
project_image_159
project_image_160
project_image_161
project_image_162

⚙️ Second platform:

project_image_163
project_image_164
project_image_165
project_image_166
project_image_167
project_image_168
project_image_169
project_image_170
project_image_171
project_image_172
project_image_173

Step 2.2: Designing product transportation mechanism

After completing both rotating platform systems, I started to work on the industrial-level transportation mechanism to move the selected sample product from the first platform to the second platform.

Conforming with my mechanical part design principle for the moving parts of the shipping workstation, I utilized gears to move the carrier on the transportation road. Nonetheless, since the product transportation mechanism requires linear motion, I designed a rack and pinion system converting rotational motion to linear motion.

#️⃣ First, I designed the transportation road, bridging the first platform with the second platform. I integrated two linear gears (racks) at the bottom of the transportation road.

project_image_174
project_image_175

#️⃣ Then, I designed pinions, the pinion connection pin, and the stepper motor direction gear.

project_image_176
project_image_177
project_image_178
project_image_179
project_image_180
project_image_181
project_image_182
project_image_183

#️⃣ After completing the rack and pinion system, I designed the transportation carrier. I employed custom bearings to connect the carrier, pinions, and the pinion connection pin to enable linear motion while maintaining a stable torque distribution.

#️⃣ Then, I designed a basic carrier arm to hold the sample product still while pulling and pushing it on the transportation road.

project_image_184
project_image_185
project_image_186
project_image_187
project_image_188
project_image_189
project_image_190
project_image_191
project_image_192
project_image_193
project_image_194
project_image_195

#️⃣ In this regard, the first Nema 17 stepper motor attached to the carrier drives pinions and the second one drives the carrier arm.

project_image_196
project_image_197
project_image_198
project_image_199
project_image_200
project_image_201
project_image_202

Step 2.3: Designing complementary accessories

After completing mechanical 3D parts for the moving workstation components, I started to design complementary accessories, including the platform roofs, for the first and second platforms.

#️⃣ Since the first platform utilizes the IR break-beam sensor as the homing method to synchronize the 200-step per rotation pattern for 60° turns, I designed the first platform roof compatible with the IR sensor receiver and transmitter.

#️⃣ Then, I designed add-ons for Raspberry Pi 5 and a USB webcam since the first platform stores the sample products for automated selection and transportation process.

project_image_203
project_image_204
project_image_205
project_image_206
project_image_207
project_image_208
project_image_209
project_image_210
project_image_211
project_image_212

#️⃣ Since the second platform employs the micro switch as the homing method to align the face separator toward the transportation road, I designed the second platform roof compatible with the micro switch.

#️⃣ Then, I designed the add-on for the thermal printer and the mount for the PCB encasement since the second platform exhibits the selected and transported product.

project_image_213
project_image_214
project_image_215
project_image_216
project_image_217
project_image_218
project_image_219
project_image_220
project_image_221
project_image_222
project_image_223

Step 2.4: Designing customized sample products

After completing the shipping workstation 3D parts, I focused on designing unique sample products since I wanted to examine the precision and efficiency of an object detection model trained on synthetic images of products that do not exist in the market.

In this regard, I was able to investigate whether it is feasible and cost-effective to initiate developing an AI-based solution for industrial operations with a synthetic data set generated from virtual product representations even before manufacturing or mass producing them.

#️⃣ Compatible with the platform face separators, I designed multipart enamel pin-inspired 3D models as virtual sample products representing these objects:

  • Wrench
  • Mouse
  • Basketball
  • Teacup
  • Hammer
  • Screwdriver
project_image_224
project_image_225
project_image_226
project_image_227
project_image_228

Step 3.0: Setting up the NVIDIA Omniverse Launcher

NVIDIA Omniverse™ is a versatile and developer-friendly platform integrating OpenUSD (Universal Scene Description) and NVIDIA RTX™ rendering technologies into existing software tools and simulation workflows with officially supported APIs, SDKs, and services. In this regard, NVIDIA Omniverse provides all the necessary building tools to envision and realize large-scale and AI-enabled virtual worlds.

Since NVIDIA Omniverse is a platform optimized for industrial digitalization and physical AI simulation and provides lots of easy-to-use tools for 3D world (environment) modeling, I decided to capitalize on its enhanced simulation and rendering features while building my shipping workstation digital twin. As NVIDIA states, various enterprises employ Omniverse's state-of-the-art services to develop digital twins as testing grounds to design, simulate, operate, and optimize their products and production facilities.

Even though NVIDIA Omniverse provides developers with the NVIDIA Omniverse Kit SDK to build OpenUSD-native applications and extensions for specific tasks, I decided to utilize the Omniverse Launcher as a single-user workstation, which gives access to all Omniverse services required to build my physically accurate shipping workstation digital twin.

#️⃣ First, install the Omniverse Launcher here.

#️⃣ Then, create an NVIDIA account and confirm the license agreement to initiate the single-user workstation.

project_image_229
project_image_230
project_image_231

#️⃣ Assign paths to store the necessary Omniverse Launcher information locally.

project_image_232

#️⃣ Since the Omniverse Launcher requires a Nucleus Collaboration Server to access all available apps, services, and assets, create a local Nucleus server and its administration account.

project_image_233
project_image_234
project_image_235
project_image_236
project_image_237

#️⃣ After establishing the local Nucleus server (service), the Launcher shows all available applications, services, connectors, and content on the Exchange tab.

project_image_238

Step 3: Forming the shipping workstation digital twin on NVIDIA Omniverse USD Composer

The Omniverse USD Composer is an application built on the Omniverse Kit and provides advanced layout tools and simulation capabilities, including but not limited to NVIDIA RTX™ Renderer and physics extension, for generating visually compelling and physically accurate worlds.

Since the USD Composer allows developers to import existing assets (designs) and render large-scale scenes with user-friendly simulation tools, I decided to set up the USD Composer on the Omniverse Launcher to build my shipping workstation digital twin.

project_image_239
project_image_240

After installing the USD Composer, I started to work on producing a realistic scenery for industrial-level shipping operations.

Plausibly, NVIDIA Omniverse provides built-in asset (3D model) and material libraries for various use cases. Also, the USD Composer includes the Asset Store displaying all available high-quality 3D models from diverse third-party content libraries.

#️⃣ First, I scrutinized all available assets provided by Omniverse (default) and Sketchfab (free Creative Commons-licensed) to produce a suitable scenery, including a close replica of my standing desk.

project_image_241
project_image_242
project_image_243
project_image_244
project_image_245
project_image_246
project_image_247
project_image_248
project_image_249
project_image_250
project_image_251
project_image_252
project_image_253
project_image_254

#️⃣ Then, I designed some custom assets with the integrated Omniverse tools to finalize my shipping warehouse scenery.

project_image_255
project_image_256
project_image_257
project_image_258

#️⃣ After completing my shipping warehouse scenery, I imported the virtual shipping workstation in the OBJ format.

#️⃣ Since the Omniverse Launcher can automatically detect and assign Fusion 360 material, color, and texture configurations, the USD Composer rendered the virtual shipping workstation to produce a flawless digital twin.

project_image_259
project_image_260
project_image_261
project_image_262
project_image_263
project_image_264
project_image_265

#️⃣ To move the first rotating platform and sample products as a single object via the physics extension, I tried to group all associated models under a new Xform. However, it was not possible since these models were references from the original OBJ file.

#️⃣ To solve this issue, I saved the Omniverse stage again by utilizing the Save Flattened As option to merge all 3D models. Then, I was able to modify and group the associated models easily.

project_image_266
project_image_267
project_image_268
project_image_269
project_image_270
project_image_271
project_image_272

#️⃣ After producing the shipping workstation digital twin, I created a few cameras to survey the virtual workstation and capture synthetic sample product images effortlessly.

project_image_273
project_image_274
project_image_275
project_image_276
project_image_277
project_image_278
project_image_279
project_image_280
project_image_281
project_image_282
project_image_283
project_image_284

Step 4: Constructing a synthetic data set of customized sample products via NVIDIA Omniverse

#️⃣ After preparing the shipping workstation digital twin for synthetic data collection, I experimented with camera, lighting, and rendering configurations to create optimal conditions.

#️⃣ Then, I applied the built-in Capture Screenshot (F10) feature by activating the Capture only the 3D viewport option so as to construct my synthetic data set of unique sample products in various poses.

project_image_285
project_image_286
project_image_287
project_image_288
project_image_289
project_image_290
project_image_291
project_image_292
project_image_293
project_image_294
project_image_295
project_image_296
project_image_297
project_image_298
project_image_299
project_image_300
project_image_301
project_image_302
project_image_303
project_image_304
project_image_305
project_image_306
project_image_307
project_image_308

🖼️ Synthetic data samples:

project_image_309
project_image_310
project_image_311
project_image_312
project_image_313
project_image_314
project_image_315
project_image_316
project_image_317
project_image_318
project_image_319
project_image_320
project_image_321

Step 5: Setting up LAMP web server, Edge Impulse CLI, and Linux Python SDK on Raspberry Pi 5

After constructing my synthetic data set, I was going to build my object detection model before proceeding with real-world shipping workstation preparations. However, while trying to upload my synthetic data samples generated by NVIDIA Omniverse USD Composer, I noticed most of them were refused by the Edge Impulse data uploader by being tagged as duplicated. I even attempted to upload six individual samples for each product, nonetheless, the issue still resumed. Thus, I decided to set up Raspberry Pi 5 earlier to perform the tasks required by the real-world shipping workstation and upload samples directly.

#️⃣ First, I installed the Raspberry Pi 5-compatible operating system image on a microSD card and initiated Raspberry Pi 5.

❗⚡ Note: While testing peripherals, I encountered under-voltage issues and purchased the official Raspberry Pi 5 27W USB-C power supply.

project_image_322
project_image_323
project_image_324

#️⃣ After initiating Raspberry Pi 5 successfully, I set up an Apache web server with a MariaDB database. I also installed PHP MySQL and cURL packages to host and enable the web workstation application features.

sudo apt-get install apache2 php mariadb-server php-mysql php-curl -y

project_image_325
project_image_326

#️⃣ To utilize the MariaDB database, I created a new user and followed the secure installation prompt.

sudo mysql_secure_installation

project_image_327

#️⃣ After setting up the LAMP web server, I installed the Edge Impulse CLI by following the official instructions for Raspbian OS.

#️⃣ First, I downloaded the latest Node.js version since versions older than 20.x may lead to installation issues or runtime errors.

curl -sL https://deb.nodesource.com/setup_20.x | sudo -E bash -

sudo apt-get install -y nodejs

node -v

#️⃣ Then, I installed the available CLI tools.

npm install -g edge-impulse-cli

project_image_328

#️⃣ After setting up the Edge Impulse CLI, I installed the Edge Impulse Linux Python SDK to run Edge Impulse machine learning models via Python.

❗ If you do not run a virtual environment on Pi 5, the system may throw an error while trying to install packages via pip. To simply solve this issue, you can add --break-system-packages.

sudo apt-get install libatlas-base-dev libportaudio2 libportaudiocpp0 portaudio19-dev python3-pip

sudo pip3 install pyaudio edge_impulse_linux --break-system-packages

project_image_329
project_image_330

Step 6: Building an object detection model (FOMO) w/ Edge Impulse Enterprise

Since Edge Impulse provides developer-friendly tools for advanced AI applications and supports almost every development board due to its model deployment options, I decided to utilize Edge Impulse Enterprise to build my object detection model. Also, Edge Impulse Enterprise incorporates elaborate model architectures for advanced computer vision applications and optimizes the state-of-the-art vision models for edge devices and single-board computers such as Raspberry Pi 5.

Among the diverse machine learning algorithms provided by Edge Impulse, I decided to employ FOMO (Faster Objects, More Objects) since it is a novel algorithm optimized for highly constrained devices with a brilliant heat map to bounding boxes technique.

While labeling my synthetic image samples, I simply applied the names of the represented real-world objects:

  • wrench
  • mouse
  • basketball
  • tea_cup
  • hammer
  • screwdriver

Plausibly, Edge Impulse Enterprise enables developers with advanced tools to build, optimize, and deploy each available machine learning algorithm as supported firmware for nearly any device you can think of. Therefore, after training and validating, I was able to deploy my FOMO model as a Linux (AARCH64) application (.eim) compatible with Raspberry Pi 5.

You can inspect my object detection model (FOMO) on Edge Impulse as a public project.

Step 6.1: Uploading and labeling training and testing images (samples)

#️⃣ To utilize the advanced AI tools provided by Edge Impulse, register here and create a new project.

project_image_331

As mentioned earlier, the Edge Impulse data uploader refused most of the synthetic image samples generated by the Omniverse USD Composer. Thus, I set up the Edge Impulse CLI to upload my synthetic data set from Raspberry Pi 5 to my Edge Impulse project directly.

Since the Edge Impulse CLI allows developers to override duplicate sample detection, I was able to upload all of my synthetic data set as training and testing samples without any problem.

❗ Use --category to choose the data category (training or testing) and add --allow-duplicates to override duplicate detection.

cd Projects/project_omniverse/omniverse_data_set

edge-impulse-uploader *.png --allow-duplicates

edge-impulse-uploader --category testing *.png --allow-duplicates

project_image_332
project_image_333
project_image_334
project_image_335

#️⃣ To employ the bounding box labeling tool for object detection models, go to Dashboard ➡ Project info ➡ Labeling method and select Bounding boxes (object detection).

project_image_336

After uploading my synthetic data set of unique sample products and activating the bounding box labeling tool, I started to draw bounding boxes around the target objects for each image sample.

#️⃣ Go to Data acquisition ➡ Labeling queue to access all unlabeled items (training and testing) remaining in the given data set.

#️⃣ After drawing bounding boxes around target objects, click the Save labels button to label an image sample. Then, repeat this process until all samples have at least one labeled target object.

project_image_337
project_image_338
project_image_339
project_image_340
project_image_341
project_image_342
project_image_343
project_image_344
project_image_345
project_image_346
project_image_347
project_image_348
project_image_349
project_image_350
project_image_351
project_image_352
project_image_353
project_image_354
project_image_355
project_image_356
project_image_357
project_image_358
project_image_359
project_image_360
project_image_361
project_image_362
project_image_363
project_image_364
project_image_365
project_image_366

Step 6.2: Training the FOMO model on synthetic sample product images

An impulse is a custom machine learning application processed and optimized by Edge Impulse. I created my impulse by employing the Image processing block and the Object Detection (Images) learning block.

The Image processing block optionally turns the input image format to grayscale or RGB and generates a features array from the passed raw image.

The Object Detection (Images) learning block represents the accessible machine learning algorithms to perform object detection.

#️⃣ Go to the Create impulse page, set the image dimensions to 320, select the Fit shortest axis resize mode so as to scale (resize) the given image samples precisely, and click Save Impulse.

project_image_367

#️⃣ To modify the raw features in the applicable format, go to the Image page, set the Color depth parameter as RGB, and click Save parameters.

project_image_368

#️⃣ Then, click Generate features to apply the Image processing block to training image samples.

project_image_369
project_image_370
project_image_371

#️⃣ After generating features successfully, navigate to the Object detection page and click Start training.

According to my prolonged experiments, I modified the neural network settings and architecture to achieve reliable accuracy and validity:

📌 Neural network settings:

  • Number of training cycles ➡ 75
  • Learning rate ➡ 0.010
  • Validation set size ➡ 3%

📌 Neural network architecture:

  • FOMO (Faster Objects, More Objects) MobileNetV2 0.35

After training with the given configurations, Edge Impulse evaluated the F1 score (accuracy) as 73.7% due to the modest volume of the validation set.

project_image_372
project_image_373
project_image_374

#️⃣ Since I decided to experiment with different model and simulation (render) configurations consecutively, I utilized two versions of the same model to achieve the results I wanted faster.

project_image_375

Step 6.3: Evaluating the model accuracy and deploying the validated model

By applying the given testing samples, Edge Impulse evaluated the model accuracy (precision) as 93.10%.

#️⃣ To validate the trained model, go to the Model testing page and click Classify all.

project_image_376
project_image_377
project_image_378

Then, I deployed the validated model as a fully optimized and customizable Linux (AARCH64) application (.eim).

#️⃣ Navigate to the Deployment page and search for Linux (AARCH64).

#️⃣ Choose the Quantized (int8) optimization option to get the optimum performance while running the deployed model.

#️⃣ Finally, click Build to download the model as a Linux (AARCH64) application (.eim) compatible with Raspberry Pi 5.

project_image_379
project_image_380
project_image_381

Step 7: Printing and assembling 3D parts of the virtual shipping workstation to build its real-world counterpart

After concluding my assignments with the shipping workstation digital twin on NVIDIA Omniverse USD Composer, I started to work on building its real-world counterpart.

#️⃣ First, on Autodesk Fusion 360, I exported all virtual shipping workstation 3D parts in the STL format individually.

#️⃣ Then, I sliced the exported parts in PrusaSlicer, which provides lots of groundbreaking features such as paint-on supports and height range modifiers.

#️⃣ Due to the fluctuating part dimensions, I needed to utilize my Anycubic Kobra 2 and Kobra 2 Max 3D printers simultaneously while printing parts. Thus, I applied the relative slicer settings for each printer.

⚙️ Platforms:

project_image_382
project_image_383
project_image_384
project_image_385
project_image_386
project_image_387
project_image_388
project_image_389
project_image_390
project_image_391

⚙️ Gears:

project_image_392
project_image_393
project_image_394
project_image_395

⚙️ Bearings:

project_image_396
project_image_397
project_image_398
project_image_399

⚙️ Transportation mechanism:

project_image_400
project_image_401
project_image_402
project_image_403
project_image_404
project_image_405

⚙️ Accessories:

project_image_406
project_image_407
project_image_408
project_image_409

⚙️ Sample products:

project_image_410
project_image_411
project_image_412
project_image_413
project_image_414
project_image_415
project_image_416
project_image_417
project_image_418
project_image_419
project_image_420
project_image_421
project_image_422
project_image_423

As mentioned earlier, I assigned PLA filament attributes for each virtual 3D part. I utilized the exact PLA filaments to print their real-world counterparts.

project_image_424
project_image_425
project_image_426
project_image_427
project_image_428
project_image_429

After printing all 3D parts successfully, I started to work on assembling the real-world shipping workstation.

project_image_430
project_image_431
project_image_432

#️⃣ First, I assembled all custom ball bearings.

#️⃣ To assemble one of my custom bearings, place the required number of 5 mm steel balls between the inner ring and the bottom outer ring.

#️⃣ Then, cap the placed steel balls with the top outer ring and utilize M3 screws to adjust the bearing tightness.

project_image_433
project_image_434
project_image_435
project_image_436
project_image_437
project_image_438
project_image_439
project_image_440
project_image_441
project_image_442
project_image_443
project_image_444

❗ Although all related 3D parts can be affixed via M3 screws after printing, plastic parts tend to loosen or break after a while due to friction and abrasion. Thus, I employed a well-known injection molding technique to make some connections more sturdy — M3 brass threaded inserts.

project_image_445
project_image_446
project_image_447

#️⃣ For each rotating platform, I fastened the required Nema 17 stepper motor and assembled the planetary gear mechanism consisting of a sun gear, three planet gears, a secondary stepper motor gear, and a Y-shaped planet carrier.

#️⃣ As explained earlier, I employed custom bearings to connect swiveling components to maintain stable torque distribution.

project_image_448
project_image_449
project_image_450
project_image_451
project_image_452
project_image_453
project_image_454
project_image_455
project_image_456
project_image_457
project_image_458
project_image_459
project_image_460
project_image_461
project_image_462
project_image_463
project_image_464
project_image_465
project_image_466
project_image_467
project_image_468
project_image_469
project_image_470
project_image_471
project_image_472
project_image_473
project_image_474
project_image_475
project_image_476

#️⃣ After completing the planetary gear mechanisms, I assembled the platform faces, face separators, and rotation pins respectively via M3 screws.

project_image_477
project_image_478
project_image_479
project_image_480
project_image_481
project_image_482

#️⃣ Then, I checked the straightness of the transportation road, bridging the first platform with the second platform.

project_image_483
project_image_484
project_image_485
project_image_486

#️⃣ I fastened the required Nema 17 stepper motors and assembled the transportation carrier consisting of two pinions, a pinion connection pin, a stepper motor direction gear, and a basic carrier arm.

#️⃣ As explained earlier, I employed custom bearings to connect swiveling components to maintain stable torque distribution.

#️⃣ Since the rack and pinion system has close-fitting components, I adjusted the tension of the M3 screws connecting the carrier components to reduce friction and stress.

#️⃣ After attaching the carrier arm, I fastened the micro switch via the hot glue gun to the left side of the transportation road, required for the automated homing sequence.

project_image_487
project_image_488
project_image_489
project_image_490
project_image_491
project_image_492
project_image_493
project_image_494
project_image_495
project_image_496
project_image_497
project_image_498
project_image_499
project_image_500
project_image_501
project_image_502
project_image_503
project_image_504
project_image_505
project_image_506
project_image_507
project_image_508
project_image_509
project_image_510
project_image_511
project_image_512
project_image_513

#️⃣ After completing the product transportation system, I assembled the platform roofs with their corresponding accessories.

#️⃣ As the selected homing methods, I fastened the IR break-beam sensor (receiver and transmitter) to the first platform roof and the micro switch to the second platform roof.

project_image_514
project_image_515
project_image_516
project_image_517
project_image_518
project_image_519
project_image_520
project_image_521
project_image_522
project_image_523
project_image_524
project_image_525

#️⃣ As I completed assembling all shipping workstation 3D parts, I attached Raspberry Pi 5, the USB webcam, and the tiny thermal printer to their corresponding add-ons.

#️⃣ Then, I fastened the Wall-E PCB to its encasement and finished all remaining PCB wiring, including the ATX adapter board 5V and 12V power outputs.

project_image_526
project_image_527
project_image_528
project_image_529
project_image_530
project_image_531
project_image_532

#️⃣ After affixing the Wall-E PCB successfully, I assembled all enamel pin-inspired sample products representing various objects by utilizing super glue.

project_image_533
project_image_534

#️⃣ Then, I placed all sample products into their corresponding compartments on the first rotating platform, sectioned by the first face separator.

project_image_535
project_image_536
project_image_537

#️⃣ After preparing the enamel pin-inspired sample products, I rigorously analyzed the moving parts of the real-world shipping workstation to detect mechanical aberrations.

project_image_538
project_image_539
project_image_540
project_image_541
project_image_542

❗ After a while, I noticed the first iteration of the carrier arm with 35% infill engendered tension issues, resulting in sudden torque fluctuations. Thus, I printed the second iteration by reducing the carrier arm size and infill (12%).

project_image_543

Step 8.0: Setting up Arduino Nano Matter on Arduino IDE

Before proceeding with programming Arduino Nano Matter, I needed to configure the required board settings and install the associated component libraries.

Since Nano Matter is a state-of-the-art IoT development board with the powerful MGM240S wireless module from Silicon Labs providing versatile connectivity options, enabling Matter IoT protocol and Bluetooth® Low Energy, it requires Arduino IDE 2.0+ or Arduino Cloud Editor to be programmed.

#️⃣ To enable the Silicon Labs core, go to File ➡ Preferences ➡ Additional boards manager URLs and add the official board package URL:

https://siliconlabs.github.io/arduino/package_arduinosilabs_index.json

project_image_544
project_image_545

#️⃣ To install the required board core, navigate to Tools ➡ Board ➡ Boards Manager, search for Nano Matter, and download the latest version.

project_image_546
project_image_547

#️⃣ After installing the core successfully, choose Arduino Nano Matter to upload sketches. Then, navigate to Tools ➡ Protocol stack and select BLE (Silabs) to utilize the accurate configurations for establishing BLE connections.

project_image_548
project_image_549

#️⃣ After setting up Nano Matter, I installed the libraries required to control the attached electronic components:

Adafruit-Thermal-Printer-Library | Download

Adafruit_SSD1306 | Download

Adafruit-GFX-Library | Download

❗ As I started to program Nano Matter, the Arduino IDE threw some errors regarding the Adafruit SSD1306 library due to the functions incompatible with the Silicon Labs core, including but not limited to digitalPinToInterrupt and portOutputRegister. To solve these compiling issues, I modified the Adafruit_SSD1306.cpp file by removing the highlighted lines below.

project_image_550

Step 8: Programming Arduino Nano Matter to finalize the workstation control panel

To prepare monochromatic images in order to display logos on the SSD1306 OLED screen and print bitmaps with the thermal printer, I followed the exact same process.

#️⃣ First, convert monochromatic bitmaps to compatible C data arrays by utilizing LCD Assistant.

#️⃣ Upload a monochromatic bitmap to LCD Assistant and select Vertical or Horizontal, depending on the screen type.

#️⃣ Then, create a header file (logo.h) to store all the converted C data arrays.

⭐ In the logo.h file, I defined multi-dimensional arrays to group the assigned logos and their sizes — width and height.

// Define the assigned ongoing task logo information as arrays. 
PROGMEM static const unsigned char *task_logos[] = {task_home_bits, task_not_found_bits, task_detected_bits, task_move_bits, task_delivery_bits, task_product_check_bits};
int task_widths[] = {task_home_width, task_not_found_width, task_detected_width, task_move_width, task_delivery_width, task_product_check_width};
int task_heights[] = {task_home_height, task_not_found_height, task_detected_height, task_move_height, task_delivery_height, task_product_check_height};
project_image_551
project_image_552
project_image_553

#️⃣ Since I needed to assign unique UUID sets (128-bit) to BLE services and data characteristics individually, I employed this online UUID generator.

As explained in the previous steps, the workstation performs various features requiring interconnected networking and multiple development board integrations. Thus, the described code snippets represent the different aspects of the presented code file. Please refer to the actual code files to inspect functions thoroughly.

📁 smart_shipping_workstation_mechanism_management.ino

⭐ Include the required libraries.

#include <Wire.h>
#include "Adafruit_Thermal.h"
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>

⭐ Import the logo.h file to obtain the converted C arrays.

#include "logo.h"

⭐ Declare the essential BLE service parameters and data characteristics.

⭐ Then, declare the ble_initialize_gatt_db function to initialize a GATT database, create a new GATT session, and add services with assigned data characteristics.

❗ As required by the Silicon Labs BLE protocol stack [BLE (Silabs)], while adding services with the assigned data characteristics, the associated UUID arrays must start with the least significant bit (LSB) from right to left (rightmost).

Service UUID: 9485c88d-cae7-43c6-9013-16d456e05bf6

0xf6, 0x5b, 0xe0, 0x56, 0xd4, 0x16, 0x13, 0x90, 0xc6, 0x43, 0xe7, 0xca, 0x8d, 0xc8, 0x85, 0x94

static void ble_initialize_gatt_db(){
  sl_status_t sc;
  // Create a new GATT database.
  sc = sl_bt_gattdb_new_session(&gattdb_session_id);
  app_assert_status(sc);

  // Add the Generic Access service to the GATT DB.
  const uint8_t generic_access_service_uuid[] = { 0x00, 0x18 };
  sc = sl_bt_gattdb_add_service(gattdb_session_id,
                                sl_bt_gattdb_primary_service,
                                SL_BT_GATTDB_ADVERTISED_SERVICE,
                                sizeof(generic_access_service_uuid),
                                generic_access_service_uuid,
                                &generic_access_service_handle);
  app_assert_status(sc);

  // By adding the device name characteristic to the Generic Access service, set the advertised local device name.
  const sl_bt_uuid_16_t device_name_characteristic_uuid = { .data = { 0x00, 0x2A } };
  sc = sl_bt_gattdb_add_uuid16_characteristic(gattdb_session_id,
                                              generic_access_service_handle,
                                              SL_BT_GATTDB_CHARACTERISTIC_READ,
                                              0x00,
                                              0x00,
                                              device_name_characteristic_uuid,
                                              sl_bt_gattdb_fixed_length_value,
                                              sizeof(advertised_device_name) - 1,
                                              sizeof(advertised_device_name) - 1,
                                              advertised_device_name,
                                              &device_name_characteristic_handle);
  app_assert_status(sc);

  // Initiate the Generic Access service.
  sc = sl_bt_gattdb_start_service(gattdb_session_id, generic_access_service_handle);
  app_assert_status(sc);

  /*
    NOTE: As required by the Silicon Labs BLE protocol stack [BLE (Silabs)], to add services and data characteristics, the UUID arrays must start with the least significant bit (LSB) from left to right.
  */

  // To add the main device service to the GATT DB, set the main service UUID. 
  // Service UUID: 9485c88d-cae7-43c6-9013-16d456e05bf6
  const uuid_128 blinky_service_uuid = {
    .data = { 0xf6, 0x5b, 0xe0, 0x56, 0xd4, 0x16, 0x13, 0x90, 0xc6, 0x43, 0xe7, 0xca, 0x8d, 0xc8, 0x85, 0x94 }
  };
  sc = sl_bt_gattdb_add_service(gattdb_session_id,
                                sl_bt_gattdb_primary_service,
                                SL_BT_GATTDB_ADVERTISED_SERVICE,
                                sizeof(blinky_service_uuid),
                                blinky_service_uuid.data,
                                &device_main_service_handle);
  app_assert_status(sc);
  
  // Add the order tag data characteristics (divided into four parts) to the main device service.
  // UUID: 9486c88d-cae7-43c6-9013-16d456e05bf6
  const uuid_128 order_tag_a_characteristic_uuid = {
    .data = { 0xf6, 0x5b, 0xe0, 0x56, 0xd4, 0x16, 0x13, 0x90, 0xc6, 0x43, 0xe7, 0xca, 0x8d, 0xc8, 0x86, 0x94 }
  };
  sc = sl_bt_gattdb_add_uuid128_characteristic(gattdb_session_id,
                                               device_main_service_handle,
                                               SL_BT_GATTDB_CHARACTERISTIC_READ | SL_BT_GATTDB_CHARACTERISTIC_WRITE,
                                               0x00,
                                               0x00,
                                               order_tag_a_characteristic_uuid,
                                               sl_bt_gattdb_fixed_length_value,
                                               1,                                                // max length
                                               sizeof(ble_data.default_char_init_value),         // initial value length
                                               &ble_data.default_char_init_value,                // initial value
                                               &order_tag_a_characteristic_handle);
  // UUID: 9487c88d-cae7-43c6-9013-16d456e05bf6
  const uuid_128 order_tag_b_characteristic_uuid = {
    .data = { 0xf6, 0x5b, 0xe0, 0x56, 0xd4, 0x16, 0x13, 0x90, 0xc6, 0x43, 0xe7, 0xca, 0x8d, 0xc8, 0x87, 0x94 }
  };
  sc = sl_bt_gattdb_add_uuid128_characteristic(gattdb_session_id,
                                               device_main_service_handle,
                                               SL_BT_GATTDB_CHARACTERISTIC_READ | SL_BT_GATTDB_CHARACTERISTIC_WRITE,
                                               0x00,
                                               0x00,
                                               order_tag_b_characteristic_uuid,
                                               sl_bt_gattdb_fixed_length_value,
                                               1,                                                // max length
                                               sizeof(ble_data.default_char_init_value),         // initial value length
                                               &ble_data.default_char_init_value,                // initial value
                                               &order_tag_b_characteristic_handle);
  // UUID: 9488c88d-cae7-43c6-9013-16d456e05bf6
  const uuid_128 order_tag_c_characteristic_uuid = {
    .data = { 0xf6, 0x5b, 0xe0, 0x56, 0xd4, 0x16, 0x13, 0x90, 0xc6, 0x43, 0xe7, 0xca, 0x8d, 0xc8, 0x88, 0x94 }
  };
  sc = sl_bt_gattdb_add_uuid128_characteristic(gattdb_session_id,
                                               device_main_service_handle,
                                               SL_BT_GATTDB_CHARACTERISTIC_READ | SL_BT_GATTDB_CHARACTERISTIC_WRITE,
                                               0x00,
                                               0x00,
                                               order_tag_c_characteristic_uuid,
                                               sl_bt_gattdb_fixed_length_value,
                                               1,                                                // max length
                                               sizeof(ble_data.default_char_init_value),         // initial value length
                                               &ble_data.default_char_init_value,                // initial value
                                               &order_tag_c_characteristic_handle);
  // UUID: 9489c88d-cae7-43c6-9013-16d456e05bf6
  const uuid_128 order_tag_d_characteristic_uuid = {
    .data = { 0xf6, 0x5b, 0xe0, 0x56, 0xd4, 0x16, 0x13, 0x90, 0xc6, 0x43, 0xe7, 0xca, 0x8d, 0xc8, 0x89, 0x94 }
  };
  sc = sl_bt_gattdb_add_uuid128_characteristic(gattdb_session_id,
                                               device_main_service_handle,
                                               SL_BT_GATTDB_CHARACTERISTIC_READ | SL_BT_GATTDB_CHARACTERISTIC_WRITE,
                                               0x00,
                                               0x00,
                                               order_tag_d_characteristic_uuid,
                                               sl_bt_gattdb_fixed_length_value,
                                               1,                                                // max length
                                               sizeof(ble_data.default_char_init_value),         // initial value length
                                               &ble_data.default_char_init_value,                // initial value
                                               &order_tag_d_characteristic_handle);
  // Add the workstation control characteristic to the main device service.
  // UUID: 9490c88d-cae7-43c6-9013-16d456e05bf6
  const uuid_128 workstation_control_characteristic_uuid = {
    .data = { 0xf6, 0x5b, 0xe0, 0x56, 0xd4, 0x16, 0x13, 0x90, 0xc6, 0x43, 0xe7, 0xca, 0x8d, 0xc8, 0x90, 0x94 }
  };
  sc = sl_bt_gattdb_add_uuid128_characteristic(gattdb_session_id,
                                               device_main_service_handle,
                                               SL_BT_GATTDB_CHARACTERISTIC_READ | SL_BT_GATTDB_CHARACTERISTIC_WRITE,
                                               0x00,
                                               0x00,
                                               workstation_control_characteristic_uuid,
                                               sl_bt_gattdb_fixed_length_value,
                                               1,                                                // max length
                                               sizeof(ble_data.default_char_init_value),         // initial value length
                                               &ble_data.default_char_init_value,                // initial value
                                               &workstation_control_characteristic_handle);                                                                                                                                                                                              

  // Add the station status and product characteristics to the main device service to notify the remote GATT client (central device) over BLE.
  // UUID: 9491c88d-cae7-43c6-9013-16d456e05bf6
  const uuid_128 station_status_characteristic_uuid = {
    .data = { 0xf6, 0x5b, 0xe0, 0x56, 0xd4, 0x16, 0x13, 0x90, 0xc6, 0x43, 0xe7, 0xca, 0x8d, 0xc8, 0x91, 0x94 }
  };
  sc = sl_bt_gattdb_add_uuid128_characteristic(gattdb_session_id,
                                               device_main_service_handle,
                                               SL_BT_GATTDB_CHARACTERISTIC_READ | SL_BT_GATTDB_CHARACTERISTIC_NOTIFY,
                                               0x00,
                                               0x00,
                                               station_status_characteristic_uuid,
                                               sl_bt_gattdb_fixed_length_value,
                                               1,                                                // max length
                                               sizeof(ble_data.default_char_init_value),         // initial value length
                                               &ble_data.default_char_init_value,                // initial value
                                               &station_status_characteristic_handle);
  // UUID: 9492c88d-cae7-43c6-9013-16d456e05bf6
  const uuid_128 product_status_characteristic_uuid = {
    .data = { 0xf6, 0x5b, 0xe0, 0x56, 0xd4, 0x16, 0x13, 0x90, 0xc6, 0x43, 0xe7, 0xca, 0x8d, 0xc8, 0x92, 0x94 }
  };
  sc = sl_bt_gattdb_add_uuid128_characteristic(gattdb_session_id,
                                               device_main_service_handle,
                                               SL_BT_GATTDB_CHARACTERISTIC_READ | SL_BT_GATTDB_CHARACTERISTIC_NOTIFY,
                                               0x00,
                                               0x00,
                                               product_status_characteristic_uuid,
                                               sl_bt_gattdb_fixed_length_value,
                                               1,                                                // max length
                                               sizeof(ble_data.default_char_init_value),         // initial value length
                                               &ble_data.default_char_init_value,                // initial value
                                               &product_status_characteristic_handle);                                               

  // Initiate the main device service.
  sc = sl_bt_gattdb_start_service(gattdb_session_id, device_main_service_handle);
  app_assert_status(sc);

  // Then, commit the GATT DB updates.
  sc = sl_bt_gattdb_commit(gattdb_session_id);
  app_assert_status(sc);
}

⭐ Declare the ble_start_advertising function to enable the BLE advertisement and configure advertisement parameters.

static void ble_start_advertising(){
  static uint8_t advertising_set_handle = 0xff;
  static bool init = true;
  sl_status_t sc;
  
  // For once, create an advertising set and configure the advertising interval as 100ms.
  if (init){    
    sc = sl_bt_advertiser_create_set(&advertising_set_handle);
    app_assert_status(sc);

    sc = sl_bt_advertiser_set_timing(
      advertising_set_handle,
      160,   // minimum advertisement interval (milliseconds * 1.6)
      160,   // maximum advertisement interval (milliseconds * 1.6)
      0,     // advertisement duration
      0);    // maximum number of advertisement events
    app_assert_status(sc);

    init = false;
  }

  // Generate the advertising information packet.
  sc = sl_bt_legacy_advertiser_generate_data(advertising_set_handle, sl_bt_advertiser_general_discoverable);
  app_assert_status(sc);

  // Start advertising and enable BLE connections.
  sc = sl_bt_legacy_advertiser_start(advertising_set_handle, sl_bt_advertiser_connectable_scannable);
  app_assert_status(sc);
}

⭐ Define the thermal printer object for passing commands through the hardware serial port of Nano Matter — Serial1.

Adafruit_Thermal printer(&Serial1);

⭐ Define the required Nema 17 stepper motor (attached to A4988 driver modules) configurations by creating a struct — stepper_config — so as to assign and call variables per stepper motor efficiently.

struct stepper_config{
  int _pins[4][2] = {{2, 3}, {4, 5}, {6, 7}, {8, 9}}; // (DIR, STEP)
  // Assign the required revolution and speed variables per stepper motor.
  int stepsPerRevolution[4] = {36, 64, 100, 50};
  int assigned_speed[4] = {16000, 12000, 12000, 1000};
  // Define stepper motor homing configurations.
  int home_step_number[4] = {1, 1, 3, 1};
  // Assign stepper motor tasks based on the associated part.
  int platform_first_rotation = 1;
  int platform_second_rotation = 2;
  int carrier_direction = 3;
  int carrier_arm = 4;
};

🤖 After declaring fundamental variables and configurations, I started to program all the workstation control panel features showcased in the following steps. You can view the code files on GitHub with thorough commentary.

project_image_554
project_image_555
project_image_556
project_image_557
project_image_558
project_image_559
project_image_560
project_image_561

Step 9: Developing the mobile (Android) workstation application w/ MIT APP Inventor

To capitalize on the reliable BLE connectivity enabled by Nano Matter, I decided to develop a mobile workstation application (Android) operating as the workstation interface. Thanks to the Wi-Fi connectivity natively provided by a smartphone, the mobile application also works as the proxy between the workstation control panel (based on Nano Matter) and the web workstation application.

Since I am familiar with MIT App Inventor's syntax and structure for programming Android applications, I decided to develop my mobile application with MIT App Inventor, which is an intuitive visual programming environment that allows developers to build fully functional applications.

Since this mobile application is open-source, you can download its APK file below to test the available features on a compatible smartphone. Unfortunately, since MIT App Inventor employs a blocks-based (drag-and-drop) tool to facilitate programming, I cannot provide a code file for the mobile application. Nonetheless, MIT App Inventor lets developers review or modify applications by importing project files in the AIA format.

Thus, you can inspect my mobile application design and programming process by opening this application's AIA file on your account.

#️⃣ First, create an MIT App Inventor account.

#️⃣ After downloading the project's AIA file (Smart_Shipping_Workstation.aia), import it into your account.

❗ Since MIT App Inventor does not provide functions for BLE data transmission natively, I utilized the latest version of the BluetoothLE extension to enable BLE connectivity for my mobile application. In this tutorial, you can get more information about the extension features.

project_image_562

#️⃣ After importing the project's AIA file, you can review or modify my coding blocks via the Blocks editor.

In the following steps, I will elucidate all of the accessible features of this mobile application.

project_image_563
project_image_564
project_image_565
project_image_566
project_image_567
project_image_568
project_image_569
project_image_570

After installing my mobile application on a compatible Android smartphone, I did not encounter any connectivity or data transmission problems while running the available features.

project_image_571
project_image_572

Step 10: Developing the web workstation application on Raspberry Pi 5

As discussed earlier, I decided to develop a versatile web application to manage the MariaDB database server hosted by Raspberry Pi 5, run the Edge Impulse FOMO object detection model, and transfer the detection results with the modified model resulting images.

Since I had already set up the essential firmware to develop the web application, I immediately started to configure the required web application settings on Raspberry Pi 5.

#️⃣ First, I created a new MariaDB database named shipping_workstation by utilizing the integrated terminal prompt.

sudo mysql -uroot -p

create database shipping_workstation;

GRANT ALL PRIVILEGES ON shipping_workstation.* TO 'root'@'localhost' IDENTIFIED BY '';

FLUSH PRIVILEGES;

#️⃣ Then, I created a database table (orders) with the required data fields under the new database.

use shipping_workstation;

CREATE TABLE `orders`( id int AUTO_INCREMENT PRIMARY KEY, `date` varchar(255), firstname varchar(255), lastname varchar(255), email varchar(255), address varchar(255), product_name varchar(255), product_id int, order_tag int, order_img varchar(255), order_status varchar(255) );

project_image_573
project_image_574

#️⃣ Since the application folders and files under the root folder of the Apache server are restricted, I changed permissions to enable file creation and modification while running the web application.

sudo chmod 777 /var/www/html

project_image_575

After setting up the mentioned web application configurations, I started to program the application features on Raspberry Pi 5.

📁 class.php

⭐ To bundle all functions under a specific structure, declare the workstation class.

⭐ In the append_new_order function:

⭐ Check the array keys of the passed order information to register an accurate shipping entry.

⭐ Generate a unique order tag (barcode) number — EAN-8.

⭐ Then, insert the generated shipping entry with the entry creation date into the given MariaDB database table.

	public function append_new_order($order_info){
		// Inspect the passed order information to register an accurate shipping entry.
		$firstname = array_key_exists("firstname", $order_info) ? $order_info["firstname"] : "empty";
		$lastname = array_key_exists("lastname", $order_info) ? $order_info["lastname"] : "empty";
		$email = array_key_exists("email", $order_info) ? $order_info["email"] : "empty";
		$address = array_key_exists("address", $order_info) ? $order_info["address"] : "empty";
		$product_name = array_key_exists("product_name", $order_info) ? $order_info["product_name"] : "empty";
		$product_id = array_key_exists("product_id", $order_info) ? $order_info["product_id"] : -1;
		$order_img = array_key_exists("order_img", $order_info) ? $order_info["order_img"] : "empty";
		
		// Assign a unique order tag (barcode) to the generated shipping entry — EAN-8.
		$N = 8;
		$order_tag = rand(pow(10, $N-1), (pow(10, $N)-1));
		
		// Update the order status and date.
		$date = date("Y/m/d__h:i:s");
		$order_status = "created";
		
		// Insert the generated shipping entry into the given MariaDB database table.
		$sql = "INSERT INTO `$this->database_table` (`date`, `firstname`, `lastname`, `email`, `address`, `product_name`, `product_id`, `order_img`, `order_tag`, `order_status`)
		        VALUES ('$date', '$firstname', '$lastname', '$email', '$address', '$product_name', '$product_id', '$order_img', '$order_tag', '$order_status')";
	    mysqli_query($this->conn, $sql);		
	}

⭐ In the update_order_info function, update the given column (data field) with the passed information. If the given data field is date, concatenate the new and older dates.

	public function update_order_info($column, $up_data, $order_tag){
		$sql = ($column == "date") ? "UPDATE `$this->database_table` SET $column=CONCAT($column, ',$up_data') WHERE `order_tag`=$order_tag" 
		                           : "UPDATE `$this->database_table` SET $column='$up_data' WHERE `order_tag`=$order_tag";
								   
		mysqli_query($this->conn, $sql);
	}

⭐ In the obtain_completed_orders function:

⭐ Obtain all completed order entries indicating transported and packaged products.

⭐ Remove the unnecessary array items from the retrieved shipping information.

⭐ Append the modified shipping information array to the associated multidimensional array.

⭐ Then, return the generated multidimensional array.

	public function obtain_completed_orders($order_tag){
		$order_status = [];
		// Obtain all completed order entries indicating transported and packaged products.
		$sql = "SELECT * FROM `$this->database_table` WHERE `order_status`='completed' AND `order_tag` = '$order_tag'";
		$result = mysqli_query($this->conn, $sql);
		$check = mysqli_num_rows($result);
		if($check > 0 && $row = mysqli_fetch_assoc($result)){
			// Remove the unnecessary array items from the retrieved shipping information.
			unset($row["id"], $row["product_id"], $row["order_status"]);
            // Then, append the modified shipping information to the associated array to pass the order status.
            $order_status = ["completed_order_info" => $row];			
		}else{
			$order_status = ["completed_order_info" => array("date" => "No data!")];
		}
		// Finally, return the generated completed order information (array).
		return $order_status;
	}

⭐ In the run_object_detection_model function:

⭐ Obtain the requested product information associated with the passed order tag (barcode).

⭐ Then, execute the given Python script to run an inference with the Edge Impulse FOMO object detection model to classify the sample product currently positioned under the USB webcam connected to Raspberry Pi 5.

⭐ Decode the JSON object transferred by the Python script to obtain the model resulting information.

⭐ As the Python script transfers the detected product class after running the FOMO object detection model:

⭐ If the ordered product is successfully found, enable the product transportation procedure and change the order status to completed. Then, append the order completion date and the path of the model resulting image generated by the Python script to the associated database entry.

⭐ Otherwise, enable the first rotating platform position change procedure.

❗ Since the web application executes the given Python script via the shell_exec function, it is not possible to observe debugging errors instantly like using the terminal. Thus, I appended 2>&1 to the command line in the shell_exec function to display debugging errors on the browser directly. In this regard, I was able to develop the web application way faster.

	public function run_object_detection_model($order_tag){
		$model_result = "";
		// First, obtain the requested product information associated with the passed order tag (barcode).
		$sql = "SELECT * FROM `$this->database_table` WHERE `order_tag`='$order_tag'";
		$result = mysqli_query($this->conn, $sql);
		$check = mysqli_num_rows($result);		
		if($check > 0 && $row = mysqli_fetch_assoc($result)){
			// Get the requested product name from the MariaDB database table.
			$requested_product = $row["product_name"];
			// Then, execute the given Python script to classify the product sample currently positioned under the USB camera.
			$path = str_replace("/assets", "/object_detection_model", dirname(__FILE__));
			$run_Python_script = shell_exec('sudo python3 "'.$path.'/run_FOMO_object_detection_model.py" 2>&1'); // Add 2>&1 for debugging errors directly on the browser.
			// Decode the JSON object transferred by the Python script to obtain the model resulting information.
			$model_resulting_data = json_decode($run_Python_script);
			// As the Python script transfers the detected product class after running the FOMO object detection model, check whether the ordered product was successfully found.
			if($model_resulting_data->detected_product_class == $requested_product){
				// If so, make the shipping workstation transportation request via the Android application.
				$model_result = "Model_Result%Product_Detected";
				// Change the order status to completed.
				$this->update_order_info("order_status", "completed", $order_tag);
				// Then, update the order completion date of the given order (shipping) entry.
				$date = date("Y/m/d__h:i:s");
				$this->update_order_info("date", $date, $order_tag);
				// Also, append the product confirmation (model resulting) image path generated by the Python script to the order entry.
				$this->update_order_info("order_img", $model_resulting_data->product_confirmation_image, $order_tag);
			}else{
				// Otherwise, make the shipping workstation current product position change request.
				$model_result = "Model_Result%Product_Not_Found";
			}
		}else{
			$model_result = "The passed order tag does not exist in the MariaDB database.";
		}
		// Finally, return the model detection result.
		return $model_result;
	}

📁 index.php

⭐ Check whether the obtained GET array contains the essential shipping information by comparing the GET array keys with the required data fields.

.../?i[firstname]=Kutluhan&i[lastname]=Aktar&i[email]=test@gmail.com&i[address]=Turkey&i[product_name]=wrench&i[product_id]=0

⭐ If the required information is received successfully, append a new shipping entry to the given MariaDB database table.

if(isset($_GET["i"])){
	// Check whether the obtained GET array contains the essential shipping information by comparing the GET array keys with the passed parameter names.
	$order_info = $_GET["i"];
	$check_parameters = array_diff($workstation->necessary_order_parameters, array_keys($order_info));
	// If the required information is received successfully, append a new shipping entry to the given database table.
	if(count($check_parameters) == 0){
		$workstation->append_new_order($order_info);
		echo "Database => New shipping entry registered successfully!";
	}else{
		echo "Database => Necessary shipping information is missing!";
	}
}

⭐ Run the Edge Impulse FOMO object detection model by executing the given Python script to find the requested product associated with the passed order tag.

else if(isset($_GET["run_model"]) && isset($_GET["order_tag"])){
	// Run the Edge Impulse FOMO (object detection) model on Raspberry Pi 5 via the given Python script to find the requested product associated with the passed order tag.
	$model_result = $workstation->run_object_detection_model($_GET["order_tag"]);
	echo $model_result;
}

🤖 To review the programming of the web application features showcased in the following steps, you can inspect the code files on GitHub with thorough commentary.

project_image_576
project_image_577
project_image_578
project_image_579

📁 Complete web application syntax:

  • /assets
    • /images
    • class.php
  • /object_detection_model
    • digital-twin-enabled-smart-shipping-workstation-linux-aarch64-v2.eim
    • run_FOMO_object_detection_model.py
  • /product_verification_images
    • default.jpg
  • index.php
project_image_580
project_image_581
project_image_582
project_image_583

Step 10.1: Setting up the FOMO object detection model on Raspberry Pi 5

While programming the web application, I needed to configure some settings on Raspberry Pi 5 to run the Edge Impulse FOMO object detection model successfully.

#️⃣ First, I changed the file permissions of the model file — Linux (AARCH64) application (.eim) — via the terminal to make it executable.

sudo chmod 777 /var/www/html/smart_shipping_workstation_database/object_detection_model/digital-twin-enabled-smart-shipping-workstation-linux-aarch64-v2.eim

project_image_584

Even after changing the permissions, the web application cannot access some Raspberry Pi 5 features requiring the root user (super-user) privileges while executing Python scripts, such as accessing frames captured by the attached USB webcam.

Although assigning super-user privileges to different users is a security risk, I decided to give the web application the ability to execute commands with root user privileges. In this case, it was applicable since the installed Apache server is only operating as a local development environment.

#️⃣ To change user privileges, open the sudoers file and alter it safely on the terminal.

sudo visudo

#️⃣ Since the Apache server employs www-data as the user name while accessing features, add these lines to the end of the sudoers file to enable the web application with root user privileges.

www-data ALL=NOPASSWD:ALL
project_image_585
project_image_586

#️⃣ After running the FOMO model successfully, I checked whether the web application saves the model results generated by the Python script to the given MariaDB database table successfully via the MariaDB prompt on the terminal.

show tables;

project_image_587
project_image_588

Step 10.2: Running the configured model via the Python SDK

#️⃣ First, install the pre-built OpenCV Python package.

❗ If you do not run a virtual environment on Pi 5, the system may throw an error while trying to install packages via pip. To simply solve this issue, you can add --break-system-packages.

sudo pip3 install opencv-contrib-python --break-system-packages

📁 run_FOMO_object_detection_model.py

⭐ Include the required modules.

import cv2
import numpy
from edge_impulse_linux.image import ImageImpulseRunner
import os
import json
import datetime
from time import sleep

⭐ To bundle all functions under a specific structure, declare the product_classification class.

⭐ In the __init__ function, initiate the high-quality USB webcam feed and declare the required variables.

    def __init__(self, model_file):
        # Initialize the USB high-quality camera feed.
        self.camera = cv2.VideoCapture(0)
        sleep(3)
        # Define the required variables to configure camera settings.
        self.cam_init = True
        self.frame_size = (320,320)
        # Define the required configurations to run the Edge Impulse FOMO object detection model.
        self.dir_path = os.path.dirname(os.path.realpath(__file__))
        self.model_file = os.path.join(self.dir_path, model_file)
        self.detected_class = "None"
        self.activate_model_result_send = False

⭐ In the run_inference function:

⭐ Get the latest frame captured by the high-quality USB webcam and transform the frame depending on the given model configurations.

⭐ Run an inference with the Edge Impulse FOMO object detection model to obtain the prediction (detection) results.

⭐ According to the detected label and generated bounding box coordinates, draw the bounding box with specified offsets on the passed frame to produce the model resulting image.

⭐ Then, save the produced model resulting image and enable data transfer for model results.

    def run_inference(self, show_img=False, debug=False, bb_offset={"x": 12, "y": 15}):
        # Run inference to detect unique product samples.
        with ImageImpulseRunner(self.model_file) as runner:
            try:
                # Print the information of the Edge Impulse model converted to a Linux (AARCH64) application (.eim).
                model_info = runner.init()
                if(debug): print('Loaded runner for "' + model_info['project']['owner'] + ' / ' + model_info['project']['name'] + '"')
                labels = model_info['model_parameters']['labels']
                # Get the latest captured image (frame) with the high-quality USB camera and alter the frame depending on the given model so as to run an inference.
                features, cropped = runner.get_features_from_image(self.latest_frame)
                res = runner.classify(features)
                # Obtain the prediction (detection) results for each label (class).
                if "bounding_boxes" in res["result"].keys():
                    if(debug): print('Found %d bounding boxes (%d ms.)' % (len(res["result"]["bounding_boxes"]), res['timing']['dsp'] + res['timing']['classification']))
                    # If the Edge Impulse model predicts a class successfully:
                    if(len(res["result"]["bounding_boxes"]) == 0):
                        self.detected_class = "None"
                    else:
                        for bb in res["result"]["bounding_boxes"]:
                            # According to the detected label, print the generated bounding boxes on the given frame to produce the model resulting image.
                            self.detected_class = bb['label']
                            if(debug): print('\t%s (%.2f): x=%d y=%d w=%d h=%d' % (bb['label'], bb['value'], bb['x'], bb['y'], bb['width'], bb['height']))
                            cropped = cv2.rectangle(cropped, (bb['x'] - bb_offset["x"], bb['y'] - bb_offset["y"]), (bb['x'] + bb['width'] + bb_offset["x"], bb['y'] + bb['height'] + bb_offset["y"]), (255, 0, 255), 3)
                # If requested, show the model detection image with the added bounding boxes if any on the screen.
                if(show_img):
                    cv2.imshow("Model Detection Image", cropped)
                # As detecting a class successfully, save the produced model resulting image and activate the model resulting data transfer.
                if(self.detected_class != "None"):
                    self.save_img(self.detected_class, cropped)
                    self.activate_model_result_send = True
            # Stop the running inference.    
            finally:
                if(runner):
                    runner.stop()

⭐ In the generate_model_resulting_info function, produce the model resulting information and pass the generated data packet as a JSON object.

    def generate_model_resulting_info(self):
        # Depending on the model results, produce the model resulting information.
        resulting_info = {"product_confirmation_image": "None", "detected_product_class": "None"}
        if(self.activate_model_result_send):
            resulting_info = {"product_confirmation_image": self.resulting_img_filename[1:], "detected_product_class": self.detected_class}
            self.activate_model_result_send = False
        # Then, pass the generated data packet as a JSON object.
        print(json.dumps(resulting_info))

🤖 For further inspection, you can review this code file on GitHub with thorough commentary.

project_image_589
project_image_590

Final Step: Conducting experiments with the real-world shipping workstation to validate and showcase all interconnected features

🚚📦📲 The shipping workstation control panel (Wall-E PCB) allows the user to calibrate each Nema 17 stepper motor position via the control buttons:

  • Control Button [B] ➡️ Select
  • Control Button [A] ➡️ Move Clockwise
  • Control Button [C] ➡️ Move Counter-clockwise

🚚📦📲 While calibrating positions, the workstation control panel turns the RGB LED to white.

project_image_591
project_image_592
project_image_593
project_image_594
project_image_595
project_image_596
project_image_597

🚚📦📲 The mobile workstation application (Android-compatible) enables the user to establish a BLE connection with the workstation control panel through Nano Matter.

🚚📦📲 After establishing a successful BLE connection, the mobile application gives the user permission to access the workstation interface.

project_image_598
project_image_599
project_image_600
project_image_601

🚚📦📲 If the user activates the automated homing sequence, the mobile application communicates with the workstation control panel to initiate the sequence in accordance with the selected homing methods:

🚚📦📲 The first rotating platform utilizes the IR break-beam sensor to synchronize the 200-step per rotation pattern for 60° turns via its specified rotation pin.

🚚📦📲 The second rotating platform utilizes the first micro switch to align the face separator toward the transportation road via its specified rotation pin.

🚚📦📲 The transportation carrier employs the second micro switch to position itself near the center of the transportation road.

🚚📦📲 The workstation control panel turns the RGB LED to green while performing the homing sequence and notifies the user of the sequence progress on the SSD1306 display.

project_image_602
project_image_603
project_image_604
project_image_605
project_image_606

🚚📦📲 Since the control panel and the mobile application exchange data packets continuously as the homing sequence continues, the mobile application informs the user of each completed task via the dedicated terminal.

project_image_607
project_image_608
project_image_609
project_image_610

🚚📦📲 Via the Wi-Fi connectivity natively provided by the smartphone, the mobile application communicates with the web workstation application and operates as the proxy between the web application and the workstation control panel.

🚚📦📲 In this regard, the mobile application enables the user to select one of the available products and transfer order information to the web application.

🚚📦📲 Then, the web application generates a unique order tag (barcode) number (EAN-8) and registers the received order information with the assigned order tag to the MariaDB database table hosted on Raspberry Pi 5 to generate a new shipping entry.

project_image_611
project_image_612
project_image_613
project_image_614
project_image_615
project_image_616
project_image_617

🚚📦📲 The mobile application lets the user request all pending shipping entries from the web application and review them chronologically.

project_image_618
project_image_619

🚚📦📲 After selecting a shipping entry, the mobile application reveals its unique order tag.

project_image_620

🚚📦📲 If the user activates the order confirmation process for the selected shipping record, the mobile application communicates with the web application so as to run the Edge Impulse FOMO object detection model to detect whether the sample product currently seen by the USB webcam is the ordered product of the given entry.

🚚📦📲 The workstation control panel notifies the user via the SSD1306 screen when the order confirmation process starts.

project_image_621
project_image_622
project_image_623

🚚📦📲 If the FOMO object detection model cannot find the ordered product, the web application dictates the mobile application to communicate with the workstation control panel so as to change the first rotating platform position to place the succeeding sample product under the USB webcam.

🚚📦📲 The workstation control panel product position change procedure:

  • Swivel the first platform by 60 degrees to bring the succeeding sample product under the USB webcam.

🚚📦📲 While changing the product position, the control panel turns the RGB LED to red and notifies the user via the SSD1306 screen.

🚚📦📲 After successfully changing the product position, the mobile application dictates the web application to rerun the object detection model.

project_image_624
project_image_625

🚚📦📲 Since the mobile application exchanges data packets with the web application and the workstation control panel continuously and simultaneously as the order confirmation process continues, the mobile application informs the user of each completed task performed by the web application and the control panel via their dedicated terminals.

project_image_626
project_image_627
project_image_628

🚚📦📲 Until finding the requested (ordered) product successfully, the mentioned assignments continue in a loop.

project_image_629

🚚📦📲 When the FOMO model detects the ordered product successfully, the web application dictates the mobile application to initiate the automated product transportation process.

🚚📦📲 Then, the workstation control panel moves the ordered product from the first platform to the second platform via the transportation carrier for packaging.

🚚📦📲 While carrying the ordered product, the mobile application and the control panel inform the user of the transportation progress via the dedicated terminal and the SSD1306 screen respectively.

🚚📦📲 The workstation control panel product transportation procedure:

  • Swivel the first platform by 180 degrees to prepare the requested product for transportation.
  • After completion, turn the RGB LED to blue.
  • Bring the transportation carrier near the first platform and adjust the carrier arm to pull the ordered product.
  • Move the carrier to the road midpoint while moving (pulling) the ordered product.
  • Then, rearrange the carrier position to place the ordered product towards the opposite side of the carrier.
  • Adjust the carrier arm to push the ordered product.
  • After completion, turn the RGB LED to yellow.
  • Move the carrier to the second platform while moving (pushing) the ordered product.
  • After transporting the product successfully, return the carrier to its default position.
  • Finally, swivel the second platform by 180 degrees to present the retrieved product.
  • After completion, turn the RGB LED to magenta.
project_image_630
project_image_631
project_image_632
project_image_633
project_image_634
project_image_635
project_image_636
project_image_637
project_image_638
project_image_639

🚚📦📲 After moving the ordered product successfully, the workstation control panel prints a shipping receipt with the assigned order tag in the EAN-8 barcode format. The thermal printer utilizes a sticker paper roll to make receipts fastenable to cardboard boxes.

🚚📦📲 After printing the shipping receipt, the control panel informs the user by turning the RGB LED to green. As discussed, the mobile application also informs the user of completed tasks via the dedicated terminal.

project_image_640
project_image_641
project_image_642
project_image_643
project_image_644

🚚📦📲 After concluding the order confirmation and product transportation processes successfully, the web application appends the completion date and the produced model resulting image path to the finalized shipping entry to remove it from the pending orders.

project_image_645
project_image_646

🚚📦📲 Then, the retrieved sample product becomes ready for packaging.

project_image_647
project_image_648
project_image_649
project_image_650
project_image_651
project_image_652

🚚📦📲 After packaging products and fastening shipping receipts to the boxes, the mobile application allows the user to scan order tags converted into the EAN-8 barcode format via the smartphone camera.

🚚📦📲 After scanning an order tag successfully, the mobile application enables the user to request detailed shipping information from the web application and review the associated model resulting image highlighting the ordered product via the modified bounding box.

project_image_653
project_image_654
project_image_655
project_image_656
project_image_657
project_image_658
project_image_659
project_image_660
project_image_661
project_image_662
project_image_663
project_image_664

🚚📦📲 After finalizing all assignments with the workstation control panel, the mobile application lets the user terminate the established BLE connection securely via the exit (X) button on the interface.

project_image_665
project_image_666

🖥️ For debugging, Nano Matter prints progression notifications on the serial monitor.

project_image_667

Further Discussions

By applying physically accurate digital twins to develop AI-oriented solutions for industrial-level shipping operations even before building real-world mechanisms or products, we can achieve the following:

🚚📦📲 develop adaptive product manufacturing procedures,

🚚📦📲 build secure, cost-effective, and efficient industrial facilities,

🚚📦📲 advance the precision of industrial-level shipping operations,

🚚📦📲 design safe, practical, cost-effective, and efficient real-world mechanisms,

🚚📦📲 forfend concomitant risks of assembling an industrial manufacturing system from scratch,

🚚📦📲 reduce exorbitant overhaul costs due to the lack of internal design blueprints,

🚚📦📲 review virtual components thoroughly to obtain the optimum performance from their real-world counterparts for multifaceted operations.

project_image_668

Project GitHub Repository

The project's GitHub repository provides:

  • Control panel code files (Nano Matter)
  • Mobile application AIA and APK files (Android)
  • Web application code files (Raspberry Pi 5)
  • PCB design files (Gerber)
  • 3D part design files (STL and OBJ)
  • Edge Impulse FOMO machine learning model (Linux AARCH64)

Schematics

project_image_669

Code

Select File

  • smart_shipping_workstation_mechanism_management.ino
  • class.php
  • index.php
  • run_FOMO_object_detection_model.py

Custom assets

See on other platforms