Warning: Declaration of Jetpack_IXR_Client::query() should be compatible with IXR_Client::query(...$args) in /home/clients/7267bc096562fcdb78c0ab60d3ac51fb/web/blog/wp-content/plugins/jetpack/class.jetpack-ixr-client.php on line 91
{"id":1281,"date":"2021-10-20T10:38:40","date_gmt":"2021-10-20T10:38:40","guid":{"rendered":"http:\/\/barradeau.com\/blog\/?p=1281"},"modified":"2021-10-22T09:36:28","modified_gmt":"2021-10-22T09:36:28","slug":"mo-amin-archive","status":"publish","type":"post","link":"https:\/\/barradeau.com\/blog\/?p=1281","title":{"rendered":"Mo Amin Archive"},"content":{"rendered":"

\"\"<\/p>\n

I worked on a quick project for Google Arts & Culture in partnership with the Mo Amin Foundation.
\nthe foundation opened up their archive allowing everyone to access Mohamed (‘Mo’) Amin’s legacy.
\nto be honest, I had no clue who he was so I was pretty surprised to discover his life and work.
\nhe documented Africa from the inside<\/em>, without that western patronizing filter<\/em>\u00a0I’m so used to (and bothered by).
\nso I’d like to encourage you to visit the dedicated page on GA&C: g.co\/MoAminArchive<\/u><\/a>\u00a0and learn more about the man and his work.<\/p>\n

the experiment itself lives here :\u00a0https:\/\/artsandculture.google.com\/experiment\/the-mo-amin-archive\/wAEJDXhB5ZAhoA<\/a><\/p>\n

the tech bit<\/h3>\n

the experiment boils down to displaying a wall of many images (as of today: 5893).
\nthe project had already been started and was using PixiJS<\/a>, a (mostly) 2D WebGL engine.
\nI never worked with PixiJS and\u00a0once I understood that the canonical way to use it was:<\/p>\n

import * as PIXI from \"pixijs\";\r\n\/\/then \r\nlet g = new PIXI.Graphics()<\/pre>\n
the work got much easier. Pixi is well documented and all the features I needed were there, out of the box.<\/p>\n
now, to tackle such a high number of images, you should avoid using the scene graph as much as possible: it’s inefficient and each time someone uses the addChild()<\/em> method, a kitten dies.
\nenter Instanced geometry<\/strong>, if you’re a real friend, you may have read a previous article about instanced geometries in three.js<\/a>, it’s the same, minus three.js, plus Pixi.
\nas the name suggests, instanced geometry lets you use a geometry and instantiate it many times.
\nthe biggest benefit is that no matter how many instances you create, they’ll be rendered in a single drawcall which is a good thing.<\/p>\n
the Pixi doc has a simple example<\/a>, the somewhat tricky part is building the buffer and the attributes:<\/p>\n
const buffer = new PIXI.Buffer(new Float32Array(geometry.instanceCount * (positionSize + colorSize)));\r\ngeometry.addAttribute('aIPos', buffer, positionSize, false, PIXI.TYPES.FLOAT, 4 * (positionSize + colorSize), 0, true);\r\ngeometry.addAttribute('aICol', buffer, colorSize, false, PIXI.TYPES.FLOAT, 4 * (positionSize + colorSize), 4 * positionSize, true);<\/pre>\n
more specifically, this method:<\/p>\n
geometry.addAttribute(id, buffer, size, normalized, type, stride, start, instance)<\/strong><\/p>\n
can be a bit intimidating, here’s a commented intitialisation (learning purpose code, not sure it works if copy\/pasted):<\/p>\n
import * as PIXI from \"pixi.js\";\r\n\/\/we extend a PIXI Geometry Object\r\nlet attributeLength, buffer;\r\nexport default class GridGeometry extends PIXI.Geometry {\r\n  constructor(total) {\r\n    \/\/create the geometry\r\n    super();\r\n    \/\/ add the a quad mesh (the 'blueprint', the shape that will be instantiated)\r\n    \/\/create the vertices\r\n    this.addAttribute(\"aVpos\", [0, 0, 0, 1, 1, 1, 1, 0]);\r\n    \/\/create the uvs\r\n    this.addAttribute(\"aVuv\", [0, 0, 0, 1, 1, 1, 1, 0]);\r\n    \/\/create the indices\r\n    this.addIndex([0, 1, 2, 0, 2, 3]);\r\n\r\n    \/\/ a HA! make it an instanced geometry\r\n    this.instanced = true;\r\n    \/\/ and set the instance count\r\n    this.instanceCount = total;\r\n\r\n    \/\/now the weird bit:\r\n    \/\/say we want to place each quad instance at a 2D position, \r\n    \/\/rescale it and make it to sample a source texture at a given uv\r\n    \/\/we'll need 6 attributes: X, Y, SX, SY, U and V \r\n    \/\/attribute length is \"how many floats are necessary per instance\"\r\n    attributeLength = 6;\r\n\r\n    \/\/create the buffer\r\n    \/\/a buffer is a \"blob of memory on the GPU\"\r\n    \/\/you need to specify how much room it will take\r\n    buffer = new PIXI.Buffer(new Float32Array(total * attributeLength));\r\n\r\n    \/\/create attributes within the buffer\r\n\r\n    \/\/each attribute is a 2D vector (C is for 'Component'): it's 2 floats long\r\n    const C = 2;\r\n\r\n    \/\/each attribute uses floats\r\n    const T = PIXI.TYPES.FLOAT;\r\n\r\n    \/\/this is called the \"stride\": how long is an instance attributes list\r\n    \/\/in this case, 6 floats so each attribute will be\r\n    \/\/ 1 float (4 bytes) * attributeLength apart\r\n    const S = 4 * attributeLength;\r\n    \/\/the stride is used to iterate over the whole buffer \r\n    \/\/and jump to the next instance\r\n\r\n    \/\/ offset states 'within the stride, where does each specific attribute start'\r\n    \/\/ with 2D vectors (C=2), the first offset is 0 floats, \r\n    \/\/ the second will be 2 floats, the third will be 4 floats\r\n    let O = 0;\r\n\r\n    \/*\r\n    now we have everythin we need to create our attributes\r\n    the first argument is the name we want to give it\r\n    this is how we can retrieve it within our shader\r\n    this reads: \r\n    every N stride, the first 2 values of this buffer will be called \"aIpos\"\r\n    it's a vec2 (C=2), I don't want it to be normalised, it's made of floats (T)\r\n    the next instance is 6 (S=6) floats away and the 2 values I want are located \r\n    at O within this stride. oh and it's an instanced buffer!\r\n    \/\/*\/\r\n    this.addAttribute(\"aIpos\", buffer, C, false, T, S, O, true);\r\n    \/\/yay, now we'll do the same for the scale\r\n    \/\/everything but the name and the offest are the same so,\r\n    \/\/before we create the next attribute, we need to increment the offest\r\n    O += 4 * C;\r\n    \/\/ok, now O = 2 floats\r\n    \/\/let's declare the scale\r\n    this.addAttribute(\"aIscale\", buffer, C, false, T, S, O, true);\r\n    \/\/and finally, the source uvs\r\n    \/\/increment the offset\r\n    O += 4 * C;\r\n    \/\/create the attribute\r\n    this.addAttribute(\"aIuv\", buffer, C, false, T, S, O, true);\r\n    \/\/ initialisation is done!\r\n\r\n    \/\/assign some values\r\n    for (let i = 0; i < total; i++) {\r\n      let iterator = i * attributeLength;\r\n\r\n      \/\/position\r\n      buffer.data[iterator++] = (Math.random() - 0.5) * 100;\r\n      buffer.data[iterator++] = (Math.random() - 0.5) * 100;\r\n\r\n      \/\/scale\r\n      buffer.data[iterator++] = Math.random();\r\n      buffer.data[iterator++] = Math.random();\r\n\r\n      \/\/source uvs\r\n      buffer.data[iterator++] = (i%columns) \/ columns;\r\n      buffer.data[iterator++] = ~~(i\/columns) \/ columns;\r\n    }\r\n    \/\/don't forget to update the new values!\r\n    buffer.update();\r\n  }\r\n}<\/pre>\n
hope it makes sense.<\/p>\n
Atlas generation<\/h3>\n
now, before playing with our instances, we’ll\u00a0 need an atlas<\/strong>. an atlas is a texture that contains many smaller pictures, it’s memory efficient and PixiJS massively uses it under the hood.
\na GPU is a state machine that needs to be set up before rendering any object. with many objects and many textures, this set up can slow down the execution a lot, if we use a single object and a single texture, it’s much faster (the instanced geometry<\/strong> above is processed as a single object).<\/p>\n
there are tools to create such atlases, the most famous is Texture Packer<\/a>, it has a lot of nice features ( tightly pack items, create metadatas, compress textures, etc. ), in my case this was only to create very coarse thumbnails to wait for the preload of larger thumbnails.
\nI usually have a Python workflow to pre-process the data and don’t need much artistic control so I use OpenCV.
\nthe script lists the images of a folder, create a grid from a tile size and stick the images into a big square texture.<\/p>\n
here’s the python atlas script<\/a>\u00a0calling it like :<\/p>\n
python atlas.py -src source -dst atlas -n color -s 64<\/pre>\n
\"\"<\/a><\/p>\n
produces a color image\u00a0 with 64px RGB thumbnails next to each other<\/p>\n
python atlas.py -src source -dst atlas -n packed -s 64 -p true<\/pre>\n
\"\"<\/p>\n
setting the ‘pack’ flag to true converts the images to greyscale and packs them into the R,G,B channels<\/p>\n
the Mo Amin collection contains a lot of black and white pictures and the packing trick<\/em> drastically reduced the atlas size (16Mo > 2.2Mo)
\nthe downside is that the JPG compression uses all 3 channels and each channel creates very visible artifacts on the other channels.\"\"<\/a>
\nIn this case, it was aligned with the art direction of the experiment, it looks like dust and scratches and the thumbnail is here to improve the quality.
\nbut it may be something you want to avoid (saving the image as PNG solves the issue bug increases the file size…).
\nI wrote a very basic python script to determine whether a picture is color or greyscale, leaving it below for posterity:<\/p>\n
import cv2\r\nfrom skimage.metrics import structural_similarity as ssim\r\ndef isColor( img ):\r\n    gr = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\r\n    b,g,r = cv2.split(img)\r\n    score = 0\r\n    v, _ = ssim( r, gr, full=True)\r\n    score += v\r\n    v, _ = ssim( g, gr, full=True)\r\n    score += v\r\n    v, _ = ssim( b, gr, full=True)\r\n    score += v\r\n    return score \/ 3 < 0.99<\/pre>\n
so now we have everything we need to display the wall of images!<\/p>\n
you should add a “sampler” and a “channel” attribute to the instances so that they know which texture and which channel to sample.
\nin the fragment shader:<\/p>\n
vec3 getColor( vec2 uv, int channel ){\r\n  \/\/use the rgb channels\r\n  if( channel == 0 )return texture2D(colorSampler, uv).rgb;\r\n  \/\/ or sample a given channel\r\n  vec4 tex = texture2D(greayscaleSampler, uv);\r\n  if( channel == 1 )return vec3( tex.r );\r\n  if( channel == 2 )return vec3( tex.g );\r\n  if( channel == 3 )return vec3( tex.b );\r\n}<\/pre>\n
<\/h3>\n
Lens effect<\/h3>\n
there was a (rather gimmicky) lens effect to implement, I was surprised how concise the code was with Pixi (three.js is a bit more verbose when it comes to postprocessing).<\/p>\n
import * as PIXI from \"pixi.js\";\r\nimport vertexSrc from \".\/lens-vs.glsl\";\r\nimport fragmentSrc from \".\/lens-fs.glsl\";\r\nlet renderer, target, rt, uniforms;\r\nexport default class Lens extends PIXI.Mesh {\r\n  constructor(_renderer, _target) {\r\n    renderer = _renderer;\r\n    target = _target; \/\/in doubt pass the stage :)    \r\n    rt = new PIXI.RenderTexture.create({ width: 256, height: 256 });\r\n    rt.scaleMode = PIXI.SCALE_MODES.NEAREST;\r\n    renderer.render(target, rt);\r\n\r\n    const geometry = new PIXI.Geometry();\r\n    geometry.addAttribute(\"position\", [0, 0, 1, 0, 1, 1, 0, 1], 2);\r\n    geometry.addAttribute(\"uv\", [0, 0, 1, 0, 1, 1, 0, 1], 2);\r\n    geometry.addIndex([0, 1, 2, 0, 2, 3]);\r\n\r\n    uniforms = {\r\n      time: 0,\r\n      uTex: rt,\r\n      radius: 150,\r\n      density: window.devicePixelRatio,\r\n      mouse: [0.5, 0.5],\r\n      resolution: [1, 1],\r\n    };\r\n    const shader = PIXI.Shader.from(vertexSrc, fragmentSrc, uniforms);\r\n    super(geometry, shader);\r\n  }\r\n\r\n  update(mouse) {\r\n    \/\/render stage to texture\r\n    this.visible = false;\r\n    renderer.render(target, rt);\r\n    uniforms.uTex = rt;\r\n    this.visible = true;\r\n    \/\/update the uniforms\r\n    uniforms.time = performance.now() * 0.0001;\r\n    let w = window.innerWidth;\r\n    let h = window.innerHeight;\r\n    uniforms.mouse[0] = mouse.x \/ w;\r\n    uniforms.mouse[1] = mouse.y \/ h;\r\n    uniforms.resolution[0] = w;\r\n    uniforms.resolution[1] = h;\r\n  }\r\n\r\n  resize(w, h) {\r\n    \/\/ 'physical' lens size\r\n    let r = Math.min(250, Math.max(w, h) \/ 4);\r\n    uniforms.radius = r;\r\n    \/\/resize render target\r\n    rt = new PIXI.RenderTexture.create({ width: w, height: h });\r\n    this.scale.set(w, h);\r\n  }\r\n}<\/pre>\n
then instantiation looks like this:<\/p>\n
let l = new Lens(app.renderer, app.stage);\r\nl.resize(window.innerWidth, window.innerHeight);\r\napp.stage.addChild(l);<\/pre>\n
the surprisingly hard part was to make it work with various pixel ratios, I may have missed something but my shader is uselessly convoluted and almost longer than the Lens class.<\/p>\n
precision mediump float;\r\nuniform sampler2D uTex;\r\nuniform float time;\r\nuniform vec2 mouse;\r\nuniform vec2 resolution;\r\nuniform float density;\r\nuniform float radius;\r\nvarying vec2 vUv;\r\nfloat lerp(float t, float a, float b) {\r\n  return a * (1. - t) + b * t;\r\n}\r\nfloat norm(float t, float a, float b) {\r\n  return (t - a) \/ (b - a);\r\n}\r\nfloat remap(float t, float a0, float b0, float a1, float b1) {\r\n  return lerp(norm(t, a0, b0), a1, b1);\r\n}\r\nvoid main() {\r\n\r\n    \/\/distance to mouse\r\n    vec2 m = mouse;\r\n    vec2 o = ( vUv - m );\r\n    vec2 dir = o;\r\n\r\n    \/\/squarify uvs\r\n    float ratio = resolution.x \/ resolution.y;\r\n    if( resolution.y > resolution.x ){\r\n        ratio = resolution.y \/ resolution.x;\r\n\t      o.y *= ratio;\r\n    }else{\r\n        o.x *= ratio;\r\n    }\r\n    \/\/ set circle scale\r\n    o \/= ratio * density * .5;\r\n    \/\/bulge\r\n    float t = remap( radius, 0., max( resolution.x, resolution.y ), 0., 1.\/density );\r\n    float l = length(o);\r\n    \/\/within the radius ?\r\n    if( l < t ){\r\n        float d = smoothstep( 0., 1.5, 1. - l );\r\n        gl_FragColor = texture2D(uTex, vUv - dir * d);\r\n    }else{\r\n        discard;\r\n    }\r\n}<\/pre>\n
by no means a canonical way of doing stuff but it works on most devices.<\/p>\n
and that’s about it, for this nice little project.<\/p>\n","protected":false},"excerpt":{"rendered":"
I worked on a quick project for Google Arts & Culture in partnership with the Mo Amin Foundation. the foundation opened up their archive allowing everyone to access Mohamed (‘Mo’) Amin’s legacy. to be honest, I had no clue who he was so I was pretty surprised to discover his life and work. he documented … [Read more…]<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":1304,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"sharing_disabled":false,"spay_email":"","jetpack_publicize_message":""},"categories":[3],"tags":[],"jetpack_featured_media_url":"https:\/\/barradeau.com\/blog\/wp-content\/uploads\/2021\/10\/brave_p6WCu89NEW.png","jetpack_publicize_connections":[],"jetpack_shortlink":"https:\/\/wp.me\/p4oXhx-kF","_links":{"self":[{"href":"https:\/\/barradeau.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1281"}],"collection":[{"href":"https:\/\/barradeau.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/barradeau.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/barradeau.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/barradeau.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1281"}],"version-history":[{"count":21,"href":"https:\/\/barradeau.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1281\/revisions"}],"predecessor-version":[{"id":1310,"href":"https:\/\/barradeau.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1281\/revisions\/1310"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/barradeau.com\/blog\/index.php?rest_route=\/wp\/v2\/media\/1304"}],"wp:attachment":[{"href":"https:\/\/barradeau.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1281"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/barradeau.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1281"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/barradeau.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1281"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}