This is a simple demonstration using Funciones de GoogleCloud and Flows to do image recognition on an image sent on Telegram.

Flows y más allá

Como desarrollador de Flows pienso a menudo quiénes son nuestros usuarios, por qué utilizan Flows, y qué necesitan para lograr sus objetivos; y luego, qué funciones tenemos que implementar para servir mejor a esos usuarios.

Flows is a powerful drag-and-drop automation engine for creating communication flows. We initially conceived it as a no-code solution, but we found many users could get really powerful behavior writing some code for specific use-cases.   These bits of code can be inside Flows, or they can be 3rd party cloud functions like AWS Lambda functions or Google Cloud Functions. 

An interesting use case: Image Recognition 

Para un ejemplo corto y divertido, te mostraré cómo implementar una app que reconozca perritos calientes. Vamos a configurar un flujo en Flows, que recibirá imágenes de los usuarios y decidirá, si enviaron un perrito caliente o no.

For many of our customers, this type of image recognition can be very powerful. Imagine you run a delivery service and you wanted to verify successful deliveries automatically. Similar to what I’m going to show, you could use location data, photos of parcels, and even recipient signatures to create a verification flow in Flows. 

Un plan para el éxito

First, we will set up a cloud function, which receives a request with a URL to an image, then it uses an image recognition API to process the image,  and responds whether there is a hotdog in the image or not.

A continuación construiremos un flujo, que recibe un mensaje de un usuario a través de un canal de mensajería (Telegram en este caso), ejecuta la función en la nube anterior, y responde al usuario si hay un perrito caliente en una foto que ha enviado.

Configuración de Google Cloud Function

First,  we will need to set up a cloud function. To get started quickly, create a cloud function using this tutorial: https://cloud.google.com/functions/docs/quickstart-console. As a ‘Trigger’ choose HTTP trigger, execution environment: Node.js 10, and in the source code field insert el fragmento de código. It’s simple code, which checks whether the request contains JSON code and answers yes or no. 

Next, you’ll need to deploy this function. To test it inside Google Cloud Platform, follow steps from the tutorial. 

‍

Para probar desde su navegador, vaya a la siguiente URL insertando la dirección específica de su función:

https://su-dirección-función.cloudfunctions.net/HotDogOrNot/?url=hello should return {“isHotDog”: true} and the address https://su-dirección-función.cloudfunctions.net/HotDogOrNot should return {“isHotDog”: false}.

¡Buen trabajo! Has creado una función en la nube de Google. Ahora tenemos que hacer nuestra función de nube más inteligente.

Configuración de la API de Google Vision

To make it smarter let’s add image recognition. For this purpose we will use the Google Vision API. To get started, follow steps 1-4 of this tutorial: https://cloud.google.com/vision/docs/quickstart-client-libraries. In the tutorial you’ll activate the Vision API and create a service account to use it.

Ahora vuelve a la función en la nube que has creado. Alterna "Variables de entorno, redes, tiempos de espera y más" y en el archivo "Cuenta de servicio" elige la cuenta de servicio VisionAPI que acabas de crear. Ahora podemos acceder a la API de Vision dentro de nuestra función.

Now let’s change the code. On a “Package.json” tab, insert este código. It will add Google Vision API library as a dependency to your function, and on "Index.js" tab update existing code with el siguiente fragmento de código.

What’s the difference compared a la previous version? We added a request to VisionAPI, which returns the etiquetas it found on the image.  Then we filter these labels by description: if it contains “hot dog” and if it has greater than 60% confidence in that label. If there is at least 1 label remaining after filtering, that means we found a hotdog on the image.

To understand how Google VisionAPI works and how the response looks like check their documentation, https://cloud.google.com/vision/docs

After that, deploy the new version of our function. To test it from your browser, find any image of a hotdog and save it's URL. Now go to URL of your function (inserting the correct address for your function) https://your-function-address.cloudfunctions.net/HotDogOrNot?url=url_to_image and replace the  “url_to_image” with a URL a la found image. If there is a hotdog in the image, the page will return {“isHotDog”: true}.

Ahora vamos a conectar esta función a Flow Builder.

Creación de un flujo en Flows

Log in into the Bird Dashboard or regístrese para obtener una cuenta if you don’t have one.

Si eres nuevo en Flows y no tienes ningún channels configurado, tendrás que ir a la página de configuración de canales, y elegir configurar el canal de Telegram. Elegí Telegram para esta demo porque es fácil y rápido de configurar.

Ahora tienes un canal que podemos usar en Flows. Ve a la página Flows , crea un nuevo flujo personalizado y elige el disparador de canal "Telegram".

Serás redirigido a una página de flujo, donde deberás elegir tu canal de Telegram como disparador, en nuestro caso es "Hotdog". Por favor, añade 2 pasos: "Obtener variables" y "Responder al mensaje del canal".

Inside the “Fetch variables” step we will call our cloud function and retrieve response into variable “isHotDog” which will contain “true” or “false” as a response from the GoogleClound function. In URL field please insert URL to your function https://your-function-address.cloudfunctions.net/HotDogOrNot and fill all other fields as on the "Fetch variable step content" picture.

And inside the “Reply to channel message” step we will respond to the customer with a message containing the yes or no response. For that insert in "Reply with message" field the following text "Hotdog on the image? {{isHotDog}}".

‍

If you have any trouble building the flow, you can export it from file: simple-hotdog-flow.json.

To test it, send an image to your Telegram bot.

So far, it looks cool! We created a small chat bot, which checks images customers sent. To make it prettier, let’s add some more steps as shown below: 

If you have any trouble building the flow, you can export it from file: flujo-hotdog-avanzado.json.

Resultados

While this is a fun example, we believe this type of functionality can be very useful for our users. 

If you want more features like this built-in in Flows, write to our support team to let us know.