Slit scan photography / video: now with body horror

The Something Awful Forums > The Finer Arts > Creative Convention > The Dorkroom > Slit scan photography / video: now with body horror

burexas.irom: Oct 29, 2007; I disapprove of what you say, and I will defend your death because you have no right to say it!

I've been doing some slit scan experiments so I thought I'd post them here as I didn't find a suitable thread for this type of experimentation.

I learned how to code in Processing for this and I'll be sharing my crappy code so you can do it yourself if so inclined. I am preparing my body for the tsunami of code citique that is sure to follow. I will go to any length not to install another Adobe product. Also many thanks to my goon brother who helped me a lot when I got stuck with coding.
https://en.wikipedia.org/wiki/Processing

Vertical temporal displacement

For starters, a basic vertical temporal displacement:

This is a small jif of the source footage:

I used two objects just to see how the different materials look. I flipped it vertically and wrote a Processing script to generate a video where each subsequent row of pixels is delayed by one frame from the previous.
Here's a video example of a similar thing (with a bit of source footage at the end):

https://www.youtube.com/watch?v=HoNZw5evQSA

One thing I learned there is that it doesn't play well with DOF, as the horizontal displacement sharpens the edges as their angle approaches the horizontal.

Also, since the minimum number of input frames for this effect is equal to the height of the video, it is advisable to pre- and post-roll for at least [height divided by framerate] seconds.

I use FFmpeg to convert the source video to a series of 8-digit named PNGs in the data folder of the Processing sketch:

code:

ffmpeg -i moviename.mov %08d.png

Do note that I'm doing all this on a Mac so results may vary.

Here is the Processing code for the vertical displacement:

code:

int firstFrame = 1; // DEFINE first frame of PNG sequence here, it will die on its own when it runs out of input
int step = 1; // DEFINE how many rows of pixels will be shifted each frame
int frameRatio = 1; // DEFINE ratio of frames to be used (e.g. 2 = every other frame)

PImage source;
PImage row;


void setup() {

  size(3840, 2160); // DEFINE size
}


void draw() {


  int i = frameCount;


  // load and draw lines from subsequent frames

  for (int j = 0; j < (height / step); j += 1) {

    println("Loading input " + String.format("%08d", Integer.parseInt(str((j * frameRatio) + firstFrame + i -1)))+".png for output frame " + str(i));

    source = loadImage (String.format("%08d", Integer.parseInt(str((j * frameRatio) + firstFrame + i -1)))+".png");

    row = source.get(0, j * step, width, step);

    image (row, 0, j * step);
  }


  // output display as PNG

  println("Saving _output" + String.format("%08d", Integer.parseInt(str(i)))+".png");

  save("_output"+String.format("%08d", Integer.parseInt(str(i)))+".png");
}

Wherever you see DEFINE in the comment is a parameter you can change depending of your input size, framerate, content or whatever.

Horizontal x/time swap

This is where things get interesting. You may have seen videos of this, as people love to do it to flowers as they get interesting colourful abstracts.
What this script does is take in a number of input frames, and then creates output frames that are number-of-input-frames wide. Height is constant.
The first output frame consists of all the first columns of all the input frames lined up horizontally left-to-right. The second frame uses all the second columns of all the input frames and so on.
The resulting number of frames cannot be more than the width of the input frames, as we swap time for width with this process. However, the resulting video can be as wide as the number of input frames available.

Here it is applied to the hand holding the racoon toy, rotated by 90 degrees:

https://www.youtube.com/watch?v=QjETmDFM0mk

The effect is a temporal "extrusion" of subsequent rotating cross sections of the original object.

But interesting things happen when shooting with a background too. Here is a jif snippet of the source footage for the next experiment:

And here is the resulting video:

https://www.youtube.com/watch?v=g3Mtr3GhW-g

I love that the static parts of the frame become horizontal rasters. The squiggly line in the beginning of the video is an awning flapping in the wind.
Right now I'm trying to find a balance between resolution / framerate / (angular) speed of subjects to make some compelling footage. As you can see, the fast moving cars become quite compact, but the person walking in the background keeps their general proportions.

And here's the code:

code:

int firstFrame = 1; // DEFINE first frame of PNG sequence here
int sourceWidth = 3840; // DEFINE width of source images - this will be the resulting number of frames
int step = 1; // DEFINE number of columns of pixels to be copied and shifted
int factorW = 64; // DEFINE factorW multiplies with factor to produce final output width, e.g. 32 * factor 120 = 4K, 16 * factor 120 = HD
int factor = 40; // DEFINE how many images to batch load and output at a time, e.g. if out of memory, halve this number and adjust factorW accordingly

int iteration = 0;
int shift = 0;


// create image arrays

PImage[] bufferFrame = new PImage[factor];
PImage[] finalFrame = new PImage[factor];


void settings() {

  size (factorW * factor, 1080); // width (product of factorW and factor) should be <= the number of input frames available
}


void setup() {

  for (int g = 0; g < factor; g += 1) {

    finalFrame[g] = createImage (width, height, RGB);
  }
}


void draw() {

  println ("---");
  println ("Framecount = " + frameCount);

  int i = frameCount;


  // load batch of factor images into bufferFrame

  println ("Loading image batch " + str(iteration + 1) + " of " + str(factorW));

  for (int j = 0; j < factor; j += 1) {

    // println ("Loading image " + String.format("%08d", Integer.parseInt(str(firstFrame + (iteration * factor) + j)))+".png");

    bufferFrame[j] = loadImage (String.format("%08d", Integer.parseInt(str(firstFrame + (iteration * factor) + j)))+".png");
  }


  // copy columns from bufferFrame to finalFrame

  println ("Starting copying");

  for (int l = 0; l < factor; l += 1) {

    for (int k = 0; k < factor; k += 1) {

      finalFrame[l].set ((k + (iteration*factor)), 0, bufferFrame[k].get (l + shift, 0, step, height));
    }
  }

  println("Stop copying");


  iteration = iteration + 1;


  // empty bufferFrame, call garbage cleaner (this is necessary because Java)

  for (int h = 0; h < factor; h += 1) {

    bufferFrame[h] = null;
  }

  System.gc();



  // save output frames

  if (iteration == factorW) {

    for (int m = 0; m < factor; m += 1) {

      image (finalFrame[m], 0, 0);

      println ("Saving _output" + str(shift) + String.format(" %08d", Integer.parseInt(str(m)))+".png");
      save ("_output"+ str(shift) + String.format(" %08d", Integer.parseInt(str(m)))+".png");
    }

    iteration = 0;

    shift = shift + factor;


    // check for exit parameter

    if (shift >= sourceWidth) {
      exit();
    }
  
  }
}

Because I don't know how to code multi-threading the code is quite slow. I have optimized it as far as to load as many images as it can and work with them from memory (loading images seems to be the chokepoint here) but the memory is easily overrun. That's what the factor variable is for, it determines how many frames to process at a time, the higher it is the faster the whole thing runs. But if you get out-of-memory errors, then reduce factor (and increase factorW accordingly). Dammit Jim, I'm a designer not a coder!

That's it for now, I'll be posting future experiments here and I hope someone else will join me.

# ? Jan 27, 2023 07:41

Adbot: ADBOT LOVES YOU

# ? Apr 27, 2024 04:22

Megabound: Oct 20, 2012

This is extremely cool and good, I'll need to sit down with the code and grok it but I think I should be able to help out with that side and do some experimentation of my own.

Super keen to see where you take this.

# ? Jan 27, 2023 12:32

tuyop: Sep 15, 2006; Every second that we're not growing BASIL is a second wasted; Fun Shoe

The video of the cars passing in the street is so wonderful and like low key psychedelic. Weird pillars on a sea of undulating colours :swoon:

Thanks for sharing your code and learning, can�t wait to give this a shot soon

# ? Jan 27, 2023 13:42

burexas.irom: Oct 29, 2007; I disapprove of what you say, and I will defend your death because you have no right to say it!

Managed to get some footage that works well with the x/time swap effect

https://www.youtube.com/watch?v=bjgYPfNWJ60

# ? Feb 4, 2023 16:22

burexas.irom: Oct 29, 2007; I disapprove of what you say, and I will defend your death because you have no right to say it!

Though I gotta agree with tuyop, it's more fun when delving into the abstract

https://www.youtube.com/watch?v=pv_zZbdyWWc

# ? Feb 4, 2023 20:58

theHUNGERian: Feb 23, 2006

burexas.irom posted:

Though I gotta agree with tuyop, it's more fun when delving into the abstract

https://www.youtube.com/watch?v=pv_zZbdyWWc

Very excellent!

# ? Feb 4, 2023 23:40

The Voice of Labor: Apr 8, 2020

what would be the hardware equivalent of the process? a huge matrix of s&h circuits?

# ? Feb 5, 2023 00:03

burexas.irom: Oct 29, 2007; I disapprove of what you say, and I will defend your death because you have no right to say it!

Not sure I can answer that with my understanding of circuits, but maybe someone else can.

To clarify a bit more, the effect is basically an expansion on the photo finish technique:

Here the film would be rolled past a slit exposing the finish line, rolling at a speed roughly relating to the speed of the contestants. Note the same "background turns into horizontal raster" effect.

My effect is basically photo finish frames taken from adjacent slits, concurrently.

In a digital camera, I guess the challenge would be to append consecutive scans of every single column of the sensor to separate files, so I would assume the solution would be either a huge memory buffer or really fast writing to storage. Or some clever circuitry, as you imlply.
An analog equivalent seems impossible, as you'd have to expose every frame of film through its own separate but adjacent slit. Maybe some variation on a fresnel lens and a huge array of rolling film frames, but at this point I'm talking out of my rear end.

# ? Feb 5, 2023 07:25

burexas.irom: Oct 29, 2007; I disapprove of what you say, and I will defend your death because you have no right to say it!

I processed some footage of water I shot while waiting for the boat.

As the process separates elements depending on their horizontal speed, it kind of separates interfering ripple patterns in the water surface.

https://www.youtube.com/watch?v=YY4uTJZ319A

I left a bit of the original footage at the end, for comparison.

# ? Feb 6, 2023 09:37

maxe: Sep 23, 2004; BLURRED SWEET STREETLIGHTS SPEEDING PAST, FAST

burexas.irom posted:

I used two objects just to see how the different materials look. I flipped it vertically and wrote a Processing script to generate a video where each subsequent row of pixels is delayed by one frame from the previous.
Here's a video example of a similar thing (with a bit of source footage at the end):

https://www.youtube.com/watch?v=HoNZw5evQSA

this poo poo is cool!

i was instantly curious what a vertical wipe would look like between the source footage and the scanned stuff so here is the quick and (very) dirty result

https://www.youtube.com/watch?v=jv8m3JFFzWw

# ? Feb 7, 2023 06:11

burexas.irom: Oct 29, 2007; I disapprove of what you say, and I will defend your death because you have no right to say it!

That's cool, and it shouldn't be too hard to implement that into the Processing code either. Also they may not be lining up ideally because I don't always output at the input framerate, mostly I just go for what looks cool.
The first process I wrote was more of a proof of concept (or maybe proof of being able to write code that actually works), so it's wholly unoptimized (it loads [height] frames for each frame separately so it spends 99.9% of its time loading images).

So, continuing to experiment with the x/time swap, I noticed that the horizontal rasters produced by static background shift very suddenly when said background has a lot of contrasting vertical lines (as in the catamaran footage where the background is all buildings) which gives them a twitchy nervous quality. Now I'm trying to get footage containing more curved or tilted perspective lines. I took some footage of a freeway junction this morning:

The resulting video is much more fluid in the background transformation.

https://www.youtube.com/watch?v=dPQcJupjdOk

I'm working with a x100v sporting the 35mm lens (or 28 with wide adapter), which is not ideal for this kind of thing. If anyone has footage they would like processed feel free to send it to me. I can't make this thing multi-thread anyway so I can spare one of my 8 cores for some processing whenever.

# ? Feb 7, 2023 09:48

gbut: Mar 28, 2008; 😤I put the UN🇺🇳 in 🎊FUN🎉

I like the abstract quality of these. It kinda messes with my mind trying to figure out the mechanics.

# ? Feb 7, 2023 19:33

burexas.irom: Oct 29, 2007; I disapprove of what you say, and I will defend your death because you have no right to say it!

Ya, it messes with my head sometimes, especially when I'm trying to predict what the outcome will be when I change the starting frame for a piece of footage.

I shot some dutch angles this morning, have some fun with perspective:

https://www.youtube.com/watch?v=0XvaEQP_AMI

https://www.youtube.com/watch?v=HBCxjSXFjMM

The weird vertical bands on this one is how I learned to program an AWB Lock button on my camera:
https://www.youtube.com/watch?v=jVZjTszxGNI

https://www.youtube.com/watch?v=XgaUR_2OESU

If you're wondering about the weird multi-colored wavy things, my town is in preparations for the carnival so there's ribbons fluttering in the wind all over the place.

# ? Feb 8, 2023 16:08

burexas.irom: Oct 29, 2007; I disapprove of what you say, and I will defend your death because you have no right to say it!

I've edited the code to do shift by rows rather than columns (so a y-time swap) and ran the catamaran footage through it, to some effect:

https://www.youtube.com/watch?v=7wj6uG8bM6U

# ? Feb 8, 2023 20:09

Adbot: ADBOT LOVES YOU

# ? Apr 27, 2024 04:22

tk: Dec 10, 2003; Nap Ghost

burexas.irom posted:

Because I don't know how to code multi-threading the code is quite slow. I have optimized it as far as to load as many images as it can and work with them from memory (loading images seems to be the chokepoint here) but the memory is easily overrun. That's what the factor variable is for, it determines how many frames to process at a time, the higher it is the faster the whole thing runs. But if you get out-of-memory errors, then reduce factor (and increase factorW accordingly). Dammit Jim, I'm a designer not a coder!

That's it for now, I'll be posting future experiments here and I hope someone else will join me.

Try changing things up so you read each image as few times as possible.

Currently, you�re holding 80 frames in memory, 40 read 40 write. If we assume a width of 400, it will take 10 batches of reads to output all 400 frames. 10x400 reads, 400 writes.

If instead you do 80 frames in memory, 1 read and 79 write. With a width of 400, it�s going to take 6 batches to output all 400 frames. 6x400 reads, 400 writes.

Should speed things up. I think. I�m not going to like set up a dev environment and try it or anything.

# ? Mar 5, 2023 06:16

The Something Awful Forums > The Finer Arts > Creative Convention > The Dorkroom > Slit scan photography / video: now with body horror