from PIL import Image
import requests

""" Step 1: Load images """ demo_image_one = Image.open( requests.get( "http://images.cocodataset.org/val2017/000000039769.jpg", stream=True ).raw )

demo_image_two = Image.open( requests.get( "http://images.cocodataset.org/test-stuff2017/000000028137.jpg", stream=True ).raw )

query_image = Image.open( requests.get( "http://images.cocodataset.org/test-stuff2017/000000028352.jpg", stream=True ).raw )

""" Step 2: Preprocessing images Details: For OpenFlamingo, we expect the image to be a torch tensor of shape batch_size x num_media x num_frames x channels x height x width. In this case batch_size = 1, num_media = 3, num_frames = 1 (this will always be one expect for video which we don't support yet), channels = 3, height = 224, width = 224. """ vision_x = [image_processor(demo_image_one).unsqueeze(0), image_processor(demo_image_two).unsqueeze(0), image_processor(query_image).unsqueeze(0)] vision_x = torch.cat(vision_x, dim=0) vision_x = vision_x.unsqueeze(1).unsqueeze(0)

""" Step 3: Preprocessing text Details: In the text we expect an special token to indicate where an image is. We also expect an <|endofchunk|> special token to indicate the end of the text portion associated with an image. """ tokenizer.padding_side = "left" # For generation padding tokens should be on the left lang_x = tokenizer( ["An image of two cats.<|endofchunk|>An image of a bathroom sink.<|endofchunk|>An image of"], return_tensors="pt", )

""" Step 4: Generate text """ generated_text = model.generate( vision_x=vision_x, lang_x=lang_x["input_ids"], attention_mask=lang_x["attention_mask"], max_new_tokens=20, num_beams=3, )

print("Generated text: ", tokenizer.decode(generated_text[0]))

方法

OpenFlamingo 是一种多模态语言模型，可用于多种任务。它在大型多模态数据集（例如 Multimodal C4）上进行训练，可用于生成以交错图像/文本为条件的文本。 例如，OpenFlamingo 可用于为图像生成标题，或根据图像和文本段落生成问题。这种方法的好处是我们能够使用上下文训练快速适应新任务。

模型架构

OpenFlamingo 寻求使用交叉注意力层来融合预训练的视觉编码器和语言模型。模型架构如下图所示。