Assignment & Intent

This project explores the creation of alternative realities through pixel manipulation, using Gianpaolo Tomseri's Mahabharata series as source material. Through threshold-based pixel sorting, I transform these narrative-rich images while diving into the colours used in these culturally significant pieces.

The technical approach uses algorithm-driven pixel sorting with five distinct modes - brightness, gold, pattern, edge, and contrast. This system allows for interactive control over sorting direction and animation speed, while carefully preserving key image elements through threshold-based processing.

In addressing photographic techniques, this method engages with light through brightness and contrast sorting, creates a sense of motion through directional sorting effects, and maintains compositional integrity via threshold-based element preservation. The visual elements are transformed through targeted manipulations: pattern detection highlights architectural details, edge enhancement emphasizes structural elements, gold detection isolates ornamental features, and contrast manipulation establishes new spatial depths.

The content and meaning of the original works remain recognizable while opening new interpretative possibilities. This approach bridges traditional imagery with digital transformation, creating a dialogue between classical storytelling and contemporary manipulation techniques. The resulting transformations can be described in three key words: "Ethereal," "Fluid," and "Transcendent," reflecting the new perspectives offered on these classical images.

Conceptual Framework

Visual Transformation

The heart of this project lies in threshold-based pixel sorting, a technique discovered while exploring the work of digital artists like Kim Asendorf. In 2010, Asendorf pioneered the "pixel sorting" effect through his Processing-based ASDFSort algorithm, which typically created dramatic streaking effects by sorting entire rows or columns of pixels.

Kim Asendorf is a digital artist known for his work with real-time generative systems and his widely recognized Pixel Sorting script.

However, for this project, I took a different approach using threshold-based sorting, which offers more nuanced control by only sorting pixels that meet specific criteria. The key distinction lies in how the sorting is applied. Traditional pixel sorting might dramatically reorganize an entire image, often resulting in the characteristic "glitch art" aesthetic.

In contrast, threshold-based sorting allows us to: - Selectively sort pixels based on specific properties - Preserve important visual elements by setting appropriate thresholds - Target particular features (like gold ornaments or architectural details) - Maintain the overall narrative structure while creating transformation.

This selective approach became particularly crucial when working with Thomasetti's Mahabharata series, where preserving narrative elements while achieving transformation was essential.

Artistic Elements & Image Analysis

In 2003, Giampaolo Thomasetti began a 12-year project creating 25 paintings capturing key moments from the Mahabharata. For this project, I focused particularly on analyzing Images 1 and 5, which showcase distinct characteristics that influenced our sorting parameters:

Giampaolo Thomasetti paintings capturing key moments from the "Mahabharata" epic.

Image 1 ("The Court Scene"):

1. Strong architectural framing with ornate pillars and archways

2. Dynamic diagonal composition with figures in motion

3. Dramatic contrast between light and shadow

4. Intricate details in clothing and architectural elements

These elements required sorting approaches that could preserve structural integrity through edge detection, enhance the diagonal flow with directional sorting, and maintain dramatic lighting by applying contrast-based sorting.

Image 5 ("The Game"):

1. Intimate indoor setting with subtle lighting

2. Horizontal composition with seated figures

3. Rich fabric textures and patterns

4. More subdued color palette with focused areas of gold ornamentation

These characteristics required pattern-based sorting to enhance textile details, gold detection to highlight ornamental elements, and careful threshold control to maintain subtle lighting gradations.

The analysis of these images directly influenced the development of our five sorting modes:

1. Brightness sorting for overall tonal control

2. Gold detection for ornamental emphasis

3. Pattern analysis for textile and architectural detail

4. Edge detection for structural preservation

5. Contrast sorting for spatial depth

Each mode was designed to respect and enhance different aspects of the original artwork while creating new visual interpretations. This careful balance between transformation and preservation became the guiding principle for the technical implementation that followed.

Technical: Simple to Complex

Foundation: Understanding Threshold-Based Sorting

Before diving into the complex sorting modes, I started to understand the fundamental concept of threshold-based pixel sorting. Unlike traditional pixel sorting that processes entire rows or columns, threshold-based sorting only reorders pixels that meet specific criteria.

Basic Concept:

1. Each pixel in an image has RGB values (0-255 for each channel)

2. We calculate a "sorting weight" for each pixel (initially just brightness)

3. We only sort pixels whose weight exceeds our threshold

4. This preserves parts of the image while transforming others

This basic implementation demonstrates several key concepts:

1. Threshold Application

FOR each column in image:
    pixels = collect_pixels_in_column()
    bright_pixels = pixels where brightness > threshold
    dim_pixels = pixels where brightness <= threshold
    
    sort(bright_pixels by brightness)
    
    write_back(bright_pixels + dim_pixels)

2. Sorting Weight Calculation

• Initially just using average RGB: (R + G + B) / 3

• This became the foundation for more complex weight calculations later

3. Preservation Through Thresholds

• Pixels below threshold remain unchanged

• Only "bright" pixels get sorted

• This maintains image structure while allowing controlled transformation

This basic version will evolve as I add:

• Different sorting modes (brightness, gold, pattern, edge, contrast)

• Various sorting directions (vertical, horizontal, diagonal)

• Smooth transitions and animation

• Interactive controls

Evolution of Sorting Parameters:

Building on our basic threshold-based sorting implementation based on birghtness, I'll show how each sorting mode utilizes the same structural framework but with different approaches to calculating pixel weights.

Contrast Sorting

CALCULATE_WEIGHT(pixel):
    # Calculate the average brightness of the pixel using its red, green, and blue values
    brightness = (R + G + B) / 3
    
    # Return the absolute difference between the brightness and 128 (middle gray)
    # This gives the "weight" as a measure of contrast from a neutral gray tone
    return |brightness - 128|

THRESHOLD_CHECK:
    # Check if the pixel's weight is above a certain threshold
    if weight > threshold then
        # If it is, include this pixel in the sorting process
        sort
    else
        # Otherwise, leave the pixel in its original position
        leave in place

Edge Detection

CALCULATE_WEIGHT(pixel):
    # Calculate the weight as the maximum difference between any two color channels (R, G, B)
    # This emphasizes edges by detecting high color contrasts between channels
    return max(
        |R - G|,  # Difference between red and green
        |G - B|,  # Difference between green and blue
        |B - R|   # Difference between blue and red
    )

THRESHOLD_CHECK:
    # Check if the pixel's weight is above a specified threshold
    if weight > threshold then
        # If it is, the pixel likely belongs to an edge due to high color contrast, so sort it
        sort
    else
        # Otherwise, leave the pixel in its original position
        leave in place

Pattern Based

CALCULATE_WEIGHT(pixel):
    # Calculate the weight as the sum of absolute differences between color channels
    # This emphasizes patterns or textures by capturing overall color variation
    return |R - G| + |G - B| + |B - R|  # Sum of color differences

THRESHOLD_CHECK:
    # Check if the pixel's weight is above a certain threshold
    if weight > threshold then
        # If it is, the pixel has significant color variation, indicating a pattern; sort it
        sort
    else
        # Otherwise, leave the pixel in its original position
        leave in place

Gold Detection

CALCULATE_WEIGHT(pixel):
    # Check if the pixel has the characteristics of a gold-like color: R > G > B
    if R > G > B then  # This ordering suggests a gold hue
        # Calculate weight with a custom formula, emphasizing red and green values over blue
        return (R * 0.5 + G * 0.3 - B * 0.2)
    else
        # If the color does not meet the gold characteristic, set weight to 0
        return 0

THRESHOLD_CHECK:
    # Check if the pixel's weight is above the threshold
    if weight > threshold then
        # If so, the pixel is likely gold, so include it in sorting
        sort
    else
        # Otherwise, leave the pixel in its original place
        leave in place

Advanced Features

What started as a simple threshold-based sorting implementation evolved into a dynamic art tool through several key features:

1. Directional Flexibility

SORT_DIRECTION:
    if direction is VERTICAL:
        sort_pixels_in_column(x)
    else if direction is HORIZONTAL:
        sort_pixels_in_row(y)
    else if direction is DIAGONAL:
        sort_pixels_in_diagonal(x + y)

The ability to sort in different directions expands artistic possibilities. Vertical sorting emphasizes height and architectural elements, while horizontal sorting enhances landscapes and seated figures. Diagonal sorting introduces dynamic flow across the composition, guiding the viewer's eye and adding movement. These techniques provide artists with versatile tools for visual storytelling.

2. Smooth Transitions

Rather than abrupt pixel movements, we implemented smooth transitions:

ANIMATION_CONTROL:
    // Calculate the progress of the animation based on frameCount and amplitude (amp)
    progress = (1 - sin(PI/2 + frameCount/amp)) / 2
    
    // Apply an easing function to the progress for smoother transitions
    eased_progress = ease(progress)
    
EASE_FUNCTION(p):
    // This function applies a cubic easing transition to the input value 'p'
    // 'transitionEase' determines the weighting of the cubic easing
    return transitionEase * p³ + (1 - transitionEase) * p

This approach creates gradual pixel transitions, enhancing the overall visual flow and providing a more pleasing aesthetic. It also improves the visualization of the sorting process, allowing viewers to better comprehend the changes occurring within the animation. Additionally, it offers control over the animation speed through the 'amp' parameter, enabling fine-tuning of the pacing for a more tailored experience.

3. Interactive Controls and Layered Effects

This system allows for experimental combination of effects:

HANDLE_CONTROLS:
    if KEY_PRESSED:
        switch(key):
            case 'V': toggle_direction()
            case 'G': enable_gold_sort()
            case 'P': enable_pattern_sort()
            case 'C': enable_contrast_sort()
            case 'E': enable_edge_sort()
            case 'B': enable_brightness_sort()
            
    if MOUSE_PRESSED:
        toggle_processing()

This control system allows for real-time switching between sorting modes and the combination of different sorting effects, providing immediate visual feedback. It also includes pause/resume functionality for precise control over the animation.

Performance Optimization

Several strategies ensure smooth performance:

PERFORMANCE_HANDLING:
    // Process multiple pixels per frame
    for i = 1 to 4:
        process_current_segment()
        
    // Only update necessary pixels
    update_modified_pixels()
    
    // Track progress for efficient processing
    if segment_complete:
        move_to_next_segment()

Key optimizations include batch processing of pixels, which improves efficiency by handling multiple pixels at once rather than individually. This is complemented by efficient pixel array handling, allowing for quicker access and manipulation of pixel data. Progress tracking for large images ensures that the sorting algorithm can manage extensive datasets without lag, while smooth animation timing enhances the overall fluidity of the visual output.

The combination of these features transforms the basic sorting algorithm into a versatile artistic tool. Multiple sorting modes can be applied sequentially, allowing for complex visual narratives. Different sorting directions create varied visual effects, adding depth and interest to the animation. Smooth transitions make the process visually engaging, and interactive controls enable users to experiment with different approaches, fostering creativity and exploration within the sorting experience.

Final Outcome

The final outcome is an interactive image processor that enables real-time manipulation of images using various threshold-based pixel sorting techniques. Users can explore visual transformations through a range of sorting modes and interactive controls.

Core Functionality includes handling images of 892x642 pixels and switching between six images from the Mahabharata series while preserving the original state for easy resets. Sorting modes feature brightness, gold detection, pattern enhancement, edge detection, and contrast sorting based on pixel luminosity and color characteristics. Directional controls allow for vertical (default), horizontal, and diagonal sorting with real-time direction switching.

Interactive controls enable users to start or stop processing via mouse clicks and use keyboard shortcuts for toggling sorting modes and images. Visual feedback includes smooth transitions, a progress bar, and console logging of current settings. This setup allows users to experiment with sorting modes, apply multiple passes with varied parameters, and create diverse visual effects. By offering start/stop control, the tool fosters an environment for exploring how different sorting parameters influence image composition while maintaining narrative elements.

<link to final file>

Below is the final Pseudo Code :

// GLOBAL VARIABLES AND SETUP
INITIALIZE:
    images[] = load_6_images()          // Mahabharata series images
    current_image = images[0].copy()    // Working image
    original_image = images[0].copy()   // Reference copy
    processing_index = 0                // Track current column/row
    is_processing = false               // Processing state
    amp = 130                          // Animation speed (higher = slower)
    sort_mode = 'brightness'           // Current sorting mode
    sort_direction = 'vertical'        // Current sort direction
    transition_ease = 0.15             // Smoothing factor

// MAIN PROCESSING LOOP
DRAW_LOOP:
    IF is_processing:
        // Process multiple segments per frame for performance
        FOR i = 1 to 4:
            IF sort_direction is 'vertical':
                process_vertical_sort()
            ELSE IF sort_direction is 'horizontal':
                process_horizontal_sort()
            ELSE:
                process_diagonal_sort()

        update_pixels()
        
    // Calculate smooth progress
    progress = (1 - sin(PI/2 + frameCount/amp)) / 2
    eased_progress = ease(progress)
    
    display_image()
    draw_progress_bar(eased_progress)

// SORTING IMPLEMENTATIONS
PROCESS_VERTICAL_SORT:
    IF processing_index >= width:
        reset_processing()
        RETURN

    x = processing_index
    segment = []
    
    // Collect column pixels
    FOR y = 0 to height:
        pixel = get_pixel_at(x, y)
        segment.add(pixel)
    
    // Sort based on current mode
    sorted_segment = sort_pixels(segment)
    
    // Write back sorted pixels
    write_pixels_to_column(x, sorted_segment)
    
    processing_index++

PROCESS_HORIZONTAL_SORT:
    // Similar to vertical but processes rows
    // Collects pixels across width instead of height

PROCESS_DIAGONAL_SORT:
    // Processes pixels along diagonal lines
    // Uses x + y for diagonal traversal

// SORTING WEIGHT CALCULATIONS
SORT_PIXELS(segment):
    // Calculate weights based on current mode
    SWITCH sort_mode:
        CASE 'brightness':
            weight = (R + G + B) / 3
        
        CASE 'gold':
            IF R > G > B:
                weight = R * 0.5 + G * 0.3 - B * 0.2
            ELSE:
                weight = 0
        
        CASE 'pattern':
            weight = |R - G| + |G - B| + |B - R|
        
        CASE 'contrast':
            brightness = (R + G + B) / 3
            weight = |brightness - 128|
        
        CASE 'edge':
            weight = max(|R - G|, |G - B|, |B - R|)
    
    // Apply threshold-based sorting
    above_threshold = []
    below_threshold = []
    
    FOR pixel IN segment:
        IF get_weight(pixel) > threshold:
            above_threshold.add(pixel)
        ELSE:
            below_threshold.add(pixel)
    
    // Sort pixels above threshold
    above_threshold.sort_by_weight()
    
    RETURN concatenate(above_threshold, below_threshold)

// INTERACTION HANDLING
HANDLE_KEY_PRESS:
    SWITCH key:
        CASE 'V': toggle_direction()
        CASE 'D': set_diagonal_sort()
        CASE 'G', 'P', 'C', 'E', 'B': change_sort_mode()
        CASE '1' to '6': change_image()

HANDLE_MOUSE_PRESS:
    toggle_processing()

// UTILITY FUNCTIONS
EASE(progress):
    // Cubic easing function
    RETURN transition_ease * progress³ + 
           (1 - transition_ease) * progress

LOG_STATE:
    // Output current state for debugging
    log({
        mode: sort_mode,
        direction: sort_direction,
        image_number: current_image,
        progress: progress,
        frame_rate: frame_rate
    })
```

Key aspects of the implementation:
1. Modular sorting system supporting multiple modes and directions
2. Threshold-based pixel manipulation for controlled transformation
3. Smooth transitions using easing functions
4. Interactive controls for real-time manipulation
5. Performance optimizations for handling large images
6. State tracking and debugging support

Each sorting mode uses the same structural framework but with different weight calculations, allowing for:
- Consistent behavior across modes
- Easy addition of new sorting modes
- Predictable interaction patterns
- Efficient processing

Conclusion

This project successfully transformed traditional pixel sorting into an interactive tool for artistic exploration of the Mahabharata series, achieving five distinct sorting modes and implementing threshold-based processing to preserve narrative elements while enabling smooth, controllable transformations. Key technical challenges included efficient pixel processing for large images, creating smooth transitions, developing specific detection algorithms for gold and patterns, and maintaining image integrity during multiple sorting passes. Artistically, the project preserved narrative elements while generating new visual interpretations, highlighted architectural and ornamental details, facilitated fluid transitions between original and transformed states, and encouraged experimental approaches through interactive controls.

Future Possibilities: An Image Encryption Tool

A potentially interesting application for this sorting method could be to use pixel sorting as an encryption mechanism. The concept could work like this:

Imagine an image transformed through a specific sequence of sorting operations:

ENCRYPTION:
    1. Apply gold sort to 45% completion
    2. Switch to diagonal pattern sort, complete 70%
    3. Apply horizontal edge detection to 30%
    4. Final vertical contrast sort to 85

This sequence creates a unique "key" that combines the sorting modes used, the direction of each sort, the completion percentage of each operation, and the order of operations. This multifaceted key encodes the transformation process, making it integral to both the sorting and decryption methods.

To decrypt the image, you have to apply the exact inverse sequence of the original operations. This involves matching the completion percentages and reversing the sorting directions used during the initial transformation, ensuring that the original image can be accurately restored.

This method introduces a visually interesting form of steganography, where the encrypted image takes on the appearance of an artistic transformation. The decryption key, represented by a sequence of artistic operations, adds an interesting layer to the process, making it both functional and aesthetically engaging.

References

1. Claude by Anthropic

2. Ellen for being such a great teacher

3. https://p5js.org/reference

4. Ciphrd. "Pixel Sorting on Shader Using Well-Crafted Sorting Filters (GLSL)." Ciphrd, 8 Apr. 2020

5. Asendorf, Kim. "ASDFSort." GitHub, 2010, https://github.com/kimasendorf/ASDFPixelSort

6. Reas, Casey, and Ben Fry. "Processing: A Programming Handbook for Visual Designers and Artists." MIT Press, 2014, https://processing.org/handbook/

7. Menkman, Rosa. "Glitch Art: Deviation From The Norm." Institute of Network Cultures, 2011, https://networkcultures.org/_uploads/NN%234_RosaMenkman.pdf

8. Nees, Georg. "Schotter." 1968, Computer-generated art, https://collections.vam.ac.uk/item/O221321/schotter-print-nees-georg/

9. Labbe, Jason. "Processing Pixel Sorting." 2015, https://github.com/JasonLabbe/pixel-sorting

10. Shiffman, Daniel. "Image Processing with p5.js." The Coding Train, https://thecodingtrain.com/tracks/image-processing-with-p5js

11. Nake, Frieder. "Algorithmic Art and Aesthetics." Leonardo, Vol. 48, No. 3, 2015

12. Cascone, Kim. "The Aesthetics of Failure: Post-Digital Tendencies in Contemporary Computer Music." Computer Music Journal, 2000

13. OpenProcessing - Pixel Manipulation Examples, https://openprocessing.org/browse/?q=pixel%20sorting