We usually host two talks per meetup, and meetups are scheduled on weekdays (preferably Tuesday to Thursday), around 18:00-19:00, with a 3h duration. Mondays are doable since no other meetups are competing, though folks are not used to attending meetups on Mondays.

There is the option of running a workshop, which should also be 2-3 hours long and would be hosted during the week. Let’s look at a Saturday morning if the workshop runs longer (e.g., 5 hours).

📝 Get ready!

There are two main ways to kick off the meetup organization. 

1. We already have a speaker who has volunteered for a concrete date. Sometimes we get volunteers through our call for speakers form: https://forms.gle/BFtuMPYiAahcLjdr7. Note that meetups are great for first-time speakers, so we are open to folks with no experience, and we will support them through their first talk.
2. We have already secured a host and a concrete date. 

In the first case, we will look for an office to host the talk when our speaker is available. Once we locate a host, we ask them if their teams want to share a talk.

If not, we will search for another community speaker. If we can’t find them (through our contacts), the meetup organizers usually volunteer to share a talk (ask on Slack #kcd-barcelona-org) or can point us to interesting speakers in the area. We keep our talks as a last resort/backup. We want to give the floor to the community members as much as possible.

In the second option, we will ask the host if their teams want to share a talk or two. Once we know how many talks we already have from the host, if any, we launch the search for the talks we need. Similar to what we described in the previous paragraph, tell the speakers what date the meetup is so they can factor that in.

Note that sometimes we can look at events like KCD Spain to ask speakers to repeat their talks at a meetup or at any other conference with local (Barcelona) speakers available to share the same talk.

The talk’s duration is usually between 20-45m, though it is possible to add more talks if they are lightning talks (5-15m). 

If speakers outside Barcelona offer to speak, as long as their company can help them with the travel, we 💜 to host them.

Once we have settled on the meetup host or date:

Find a host willing to host our meetup, including catering for the attendees. How to find hosts? Past sponsors willing to repeat, your own company, contacts you make attending other meetups at different companies, spread the word at meetups that we are open to hosting in various offices, etc. If a speaker offers to give a talk, ask if their company could host us.

Check the space to make sure it’s comfortable for talks. What is the capacity? We usually get 30-50 attendees, we can do events in smaller offices, and we must adjust the meetup capacity if it is smaller.

Are there any requirements to access the host’s office? We need to know if the host needs a list of names/IDs/passports. In those cases, we have to close the RSVP the day before and send the list to the host on the morning of the event.

Ensure the speakers have what they need: a screen for presentations, guest wifi, or anything else they might need (whiteboard to draw or any material in case it’s a workshop and not a talk).

Ask speakers about the talk language to make it explicit in the meetup. We favor having all talks in the same language (Spanish, English, Catalan), but we can also mix it as long as it’s clear in the event we publish.

Ask the host if they want to record the meetup and have the means to. We don’t usually record them ourselves. However, we are open to recording them if the speakers agree. In this case, if the host can record, always ask for speakers’ approval.

Once we have it all ready, always send an e-mail with cc: to hello@cloudnative.barcelona (the alias for the meetup organizers) to coordinate the meetup with hosts and speakers. Other organizers can jump in to help if needed.

Using the e-mail thread above, ask speakers for the talk title and talk description. Confirm the speakers’ roles and company with the social network account/s they want to share (LinkedIn, etc.). Ask the host for the social accounts/URLs they want us to mention (LinkedIn, their web page, etc.).Effortless

Create the meetup presentation based on this sample. We will use the first slide as the cover image for our meetup and share it with our social networks. We organize this in our private Google Drive per event, so create the folder for the event and copy & modify the presentation there.

Create the draft under https://www.meetup.com/cloud-native-bcn/events/. It’s easier to duplicate a previous event and make sure you change the location and dates. Adjust the capacity, considering that only 50% will show up (if the host has a capacity of 40, it is open to 70-80 slots). Add the question if we need to gather names and ID/passports. And check with the host if they need to list the day before. In that case, prepare the meetup so that it closes RSVPs the day before. It’s not ideal as sometimes latecomers will not see it until the day, though necessary if the host needs to provide the attendees’ list to the front desk/reception one day in advance. Make sure you enable chat, as users prefer to communicate via meetup chat, not just comments in the event page.

Share the draft in the e-mail thread with the host and speakers to ask for their feedback and adjust accordingly. Always remember to thank the host explicitly for providing food & refreshments, apart from sharing their space.

We aim to publish one month in advance (or more). If we are short in time, always mention the deadline to publish in the e-mail thread and that we need everyone’s approval to proceed. Follow up sooner if we are close to the deadline, and leave time if we are in no rush.

🎉 Publish time!

Publish the meetup once we have the approvals 🎉 Don’t forget to push the button to “Notify” your group. In this way, the group members will get the notification and can RSVP before we start sharing this more broadly.

Sometimes the host also wants to publish on their groups, and that is ok. For example: https://www.meetup.com/adevinta-spain/events/299104275/. We will not encourage this if the host has limited capacity, as it would complicate spreading the RSVPs we allow on each meetup. In this particular case, it works very well as the host has a capacity 150.

Because the CNCF uses Bevy, we have to also publish under https://community.cncf.io/cloud-native-barcelona/. You should have access to clone from a previous event and make similar changes as you did with the meetup one. Be aware that, with Bevy, we cannot ask for name/ID/passport. In those cases, publish an event with no RSVPs, like https://community.cncf.io/events/details/cncf-cloud-native-barcelona-presents-cloud-native-barcelona-adevinta/, and point attendees to RSVP at meetup.com. If the host does not require additional information, we can publish a regular event under Bevy and spread the capacity between meetup.com and Bevy, as we did with https://community.cncf.io/events/details/cncf-cloud-native-barcelona-presents-cloud-native-barcelona-meetup-de-enero-2024/. As we have 1000 users in meetup and less than 100 in Bevy, spread the capacity with that in mind, 80% meetup, 20% Bevy.

Update the e-mail thread with hosts and speakers with the event URLs. And remember to thank them 🫶!

One or two weeks before the event, we need to publish the event to our networks. We don’t do it before, as the closer we are to the event, the more probable folks who RSVP will actually come. Share on our LinkedIn group and X. E.g. https://www.linkedin.com/feed/update/urn:li:activity:7171111696596504577 and https://x.com/cloudnativebcn/status/1749510135423074572. Repost from your own accounts and ask the other organizers to give the posts some “love”. Remember to mention, when possible, the speakers and the host’s social accounts.

One or two weeks before the event, share the event on Slack. The best for Barcelona events is BncEngineering, #events-meetups-confs. Another good Slack channel for Barcelona CNCF events is #cncf-barcelona under the CNCF slack.

🙇‍♀️ Almost there!

If the host requires names/IDs/passports, regularly check meetup.com to see if users replied correctly so we can make the final list for the host before the event. Message folks on meetup.com privately if we are missing information.

A few days before the event, check that everyone is ready by updating the e-mail thread with hosts and speakers. We can ask them if they need anything from us and if it’s ok to be there 30 minutes before the event to get everything ready.

If the host needs a list of attendees one day before the event, make sure you close the RSVPs and send the list. You’ll find examples of the list in our private Google Drive. We download the data from meetup.com and send a simplified list with the host’s required data. Remember to merge the lists before sending if we also have folks in Bevy!

Keep an eye on the event chat and comments regularly, as the community might ask questions (are we recording, what the talk languages are, etc.).

🎉 Meetup day!

Make sure to check any messages we have in the meetup, there are some users asking questions last minute (“I can’t locate the venue”, “I did not RSVP in time, and I want to join”, “I can’t come”, etc.). Try to be responsive.

Arrive at the venue 30 minutes before (or ask other organizers to be there if you can’t). We need to help the host set up and help the speakers check their setup.

Introduce the meetup using the presentation created from the sample. Next, give the floor to the host to present their company. And finally, present the 🔝 speakers 🎙️.

Take pictures from each presentation, including ours, the host, and the speakers. Upload them to the meetup. And some from the networking part. E.g. https://www.meetup.com/cloud-native-bcn/photos/34124873/.

It’s nice to share with our networks while the meetup is ongoing. E.g. https://x.com/cloudnativebcn/status/1768356599859445926. Remember to mention the host and speakers if possible.

Stay until the end and help the host tear down.

🙌 Wrap up!

Update the e-mail thread with hosts and speakers to thank them for a successful event, and hoping we can collaborate in future events. If the host recorded the meetup, ask for the recording so we can share it in our meetup.com event (add it to the comments), and use it in our final social post. We can do a final LinkedIn post (or Twitter), share the recording, and thank speakers and hosts again.

Make sure you upload all photos to the meetup.com event for future reference. E.g. https://www.meetup.com/cloud-native-bcn/photos/34124873/

If attendees ask for presentation slides, follow up with the speakers to share them in the meetup comments.

And don’t forget to enjoy meetup organization!

🙌 What I Accomplished: Community & Connection

Joining Grafan Lab’s developer advocacy team meant diving headfirst into a dynamic ecosystem. Here are some key highlights:

• Community building defined my year! I co-organized a whopping 29 meetups, with 13 dedicated to Grafana & Friends. I am most proud of:
  • Contribute to several thriving local communities: Software Crafters Barcelona, Cloud Native Barcelona, Grafana & Friends Barcelona.
  • Co-organizing not one but two Global Day of Code Retreat in November.
  • Being constant in providing a recurrent space for Coding Dojos every two weeks for the whole year!
  • I supported TechFems as they embarked on their new journey to empower women+ in tech at the end of 2023. The volunteer team is most impressive and inspiring!
  • Helping revive the Cloud Native Barcelona meetups. It was a pleasure to kick things off with the CNCF Cloud Native Sustainability week. And I loved joining the Kubernetes Community Days Spain team.
  • A big shoutout to all the fantastic hosts, co-organizers, and attendees who made this possible. It takes a city!
• Inspiration. I repeated as a judge for Technovation Girls. So many girls gather worldwide to tackle the UN 17 Sustainable Development goals! Their projects are an inspiration.
• Grafana contributions. I focused on supporting the vibrant Grafana k6 community in the forums. I felt appreciated and learned immensely from users and our skilled and patient k6 core developers. I also contributed:
  • Leading the k6 and Grafana forum unification.
  • Documentation updates.
  • I dabbled in minor code contributions in … Typescript! Learning JavaScript on the job was a bonus!
• Expanding my Network. I enjoyed several conferences: BilboStack, KubeCon & Cloud Native Europe, Fosdem, Spring I/O Barcelona, Software Crafters Barcelona (volunteering on our 10th anniversary 🎂), and DevOps Barcelona. This allowed me to meet developers, other advocates, and allies while reconnecting with old friends.
• Growing With a Supportive Team. Last but not least, a huge thanks to the fantastic developer advocates team at Grafana. Their support was vital on this new path: David & Pepe, Melori, Paul, Nicole, Leandro, Marie, Matt, Usman, and Antonio. And to my favorite tech writer, Matt, who has unwavering patience with tech writer newbies like me and for sharing his knowledge.

On a more personal level, this year marked the final return to normal after the pandemic. Not only were community meetups back with regular attendance levels, but I was also able to get back on the road and explore new hobbies:

• I enjoyed Bruce Springsteen & the E-Street Band concerts in Washington and Barcelona. They were finally back on the road!
• Explored new cities like Nice, Antibes, Monaco, and Oporto.
• I visited Madrid a few times throughout the year. My favorite one is to meet Elastic support engineers at their Support Summit!
• I indulged in some Lego-building therapy.
• I started cycling regularly in Barcelona with Kleta while doing AquaFitness 5 days a week!
• And celebrated the end of the year with the coveted 1-year Grafana Labs mug.

🙇‍♀️ What I Wasn’t Able to Achieve: Lessons Learned

Some unforeseen circumstances brought their own set of challenges in 2023:

• Health Hurdles. Navigating health issues from September to December presented unexpected roadblocks. This meant stepping back from commitments like TestingUnited and PulpoCon. Facing these challenges head-on became a valuable lesson in self-care and pacing. And lots of patience in navigating the public health system.
• Backup Plans Matter. While unplanned conference absences are never ideal, lesson learned! Having a co-speaker for PulpoCon ensured the workshop ran smoothly; thanks, Daniel! This experience reinforced the importance of having backup plans whenever possible. I understand the impact last-minute changes can have and that any help smoothing those is much appreciated. Being an event organizer myself, I should never forget that.
• More focus on tech writing. I improved my writing skills thanks to Matt Dodson and Google’s Tech Writing course. I wish I had taken more time to practice the writing craft. I realized I had so much to learn; consistency is the best way to continue improving.

The most important learnings from 2023 are:

• The importance of self-compassion.
• Set realistic expectations.
• Have flexibility to adjust plans.

📅 What I Will Focus on in 2024: Setting New Goals

2023 was a year of growth and discovery as I navigated the world of developer advocacy. While I achieved more than I ever anticipated, I also learned the importance of pacing myself and prioritizing my well-being.

Looking forward, I’m taking a more fluid approach to my goals for 2024. Think of these as aspirations. These are exciting possibilities I’ll pursue with flexibility and openness to change.

Community & Advocacy:

• TechFems, Technovation Girls, and Local Communities. My commitment to these initiatives remains unwavering. My core values are supporting women in tech and fostering vibrant local communities. They infuse me with so much energy that keeps me going the whole year.
• Safe Spaces for Speakers. I’m passionate about creating welcoming environments for speakers. I’ll seek opportunities to provide platforms for diverse voices, especially first-time speakers.
• Fosdem & DevRoom Management. Exciting opportunities ahead: Volunteer at Fosdem again and contribute more by managing a DevRoom. I’m eager to help in this incredible open-source event made by the community for the community.
• Connecting with local Grafanistas. Help create spaces for the Barcelona crew at Grafana to meet like HackUPC.
• CNCF Ambassador. I’ll work towards becoming a CNCF Ambassador.
  • I’ll continue to support the TAG Environmental Sustainability initiative. It aligns very well with my passion for collaboration and sustainability.
  • Co-organizing the first in-person Kubernetes Community Days Barcelona sounds like a stimulating challenge. I’m excited to contribute to this valuable event. I am forever thankful to the DevBcn team for hosting us.

Technical Growth & Skilling Up:

• Grafana Loki Journey. I will embark on a learning path to explore Grafana Loki OSS Deployment. This will expand my technical knowledge and understanding and allow me to dive deep into Grafana.
• Tech Writing. I will regularly blog and contribute to Grafana documentation and tutorials to hone my technical writing skills.
• Coding Dojos & Architectural Kata with Software Crafters Barcelona. I will continue to co-organize Coding Dojos. And I plan to co-facilitate an architectural kata around monitoring & alerting. Giving back to the Software Crafters Barcelona is fulfilling in so many ways.
• CKA Certification (Stretch). Pursuing the CKA certification would be rewarding. I’ve been meaning to for years. And I’m pretty sure this will again fall through the cracks.

Conferences, Travel & More:

• KubeCon & Cloud Native 2024 Europe. Speaking at this prestigious event is my dream. It would be a remarkable achievement. I’ll submit proposals and keep my fingers crossed!
• Women in Tech Mentoring: Supporting women in tech remains a priority. I’ll actively seek opportunities to connect and empower others. I learn so much from supporting others to succeed!
• Conferences. I plan to attend Fosdem, KubeCon & Cloud Native Europe, Google Cloud Next, PulpoCon, and DevOps Barcelona. And possibly the Mobile World Congress. It would be my first time there. Exciting possibilities!
  • I must remember to write about these experiences!
• Travel. I’ll explore Brussels, Paris, Mallorca, Lisbon, London, Amsterdam, Prague and Athens. Some for the first time!
• Reading Revival. I’m rekindling my love for reading paper books. I’ll set a goal of 12 books, 6 work-related and 6 for pure enjoyment.
• Continue cycling, AquaFitness, and add other fitness activities like Zumba.
• And, of course, some Bruce Springsteen concerts: Madrid & Prague, here I go!

I must remember these are aspirations, not rigid goals. I’ll embrace the unexpected, focus on my well-being, and celebrate every step along the way. Here’s to a year of learning, growth, and meaningful connections!

Meet Imma Valls, one of our dedicated TechFems board members, committed to changing diversity in the tech sector. She’s dedicated to supporting women and underrepresented individuals in tech, creating a welcoming space within TechFems for anyone looking to change careers or find their place in the tech world.

Recently, I had the chance to sit down with Imma at a TechFems event and let me tell you, I was genuinely inspired by her commitment to mentoring our women+ in tech. Her support is an inspiration for all of us.

Join me in this conversation, where we explore Imma’s journey, her insights on the tech scene, and the transformative power of mentorship. Let’s tap into her experiences and wisdom as she champions diversity within the tech sector.

TechFems: Can you share a bit about your background and journey in the tech industry and what inspired you to join this field?

I was born in the countryside, in a farmhouse without electricity or a phone. And water from a well! I was raised by a very conservative family where women were supposed to get married, not work outside the home, and raise a family and support their husbands and children. To this day, I still don’t understand why they allowed me to study electrical and electronics engineering. On top of that, it was also costly for my family since I had to move to Barcelona to follow my studies.

At university, I had a colossal impostor syndrome. I felt very much out of place in the big city. I was lucky with the friends I made at university, as they supported and encouraged me. Even if we were 20% of women, our group of about 20 was almost half women. And the guys in the groups were great allies. So, until I started my first job, I did not realize I was working in a men’s domain. Back then, it was relatively easy to land a job in tech if you had an engineer’s degree. And for the following 15-20 years, I almost felt like one of the guys in most teams.

Almost 6 years ago, I joined elastic.co. t was in my first genuinely diverse company. People from around the world. And the company made a real effort to support underrepresented groups in tech. For example, there was no gender gap in payment for women, and the company valued diverse viewpoints. They walk the talk. Some of my co-workers already volunteered at underrepresented groups’ communities, inspiring me to follow the same path: to help the tech industry become more welcoming to all sorts of folks.

My current company, Grafana, is similar in culture. And I love how the company supports the causes their employees care about, sponsoring when needed.

TechFems: What motivated you to join TechFems, which supports women in tech, and how has your experience been so far?

I met Henriette at one of Codebar’s events in Barcelona that my previous employer, Elastic, sponsored. I found her story inspiring: changing your career is never too late. Once Henriette started a group to support migrant women who were trying to make a career change to tech, I joined the group. And here I am.

It’s great to be in contact with people who are not as privileged as I am. It’s humbling. Even if being a minority in the team is not easy, I have been able to follow my passion and work in tech. Having an engineering degree was crucial. That path is unnecessary these days, and I love seeing how women discover what they enjoy in tech. They can get the skills via bootcamps and self-learning and then land their first job in tech. It democratizes access to a career path that can significantly impact and be very fulfilling.

TechFems: In your opinion, what unique challenges do women face in the tech industry, and how do you see this organization addressing those challenges?

Being a woman in tech means navigating an industry where we are underrepresented, it is difficult to enter, and even more challenging to stay and advance. TechFems is uniquely positioned to help women navigate the entry phase by offering coaching and knowledge sessions for students.

And, for coaches, it supports staying in tech and advancing. We realize we are not alone in tech and thrive by supporting each other.

One of the challenges for women in a male-dominated field is speaking up and being heard. TechFems provides a safe space for women to ask questions and express themselves more freely. Women-centric tech events are crucial to creating safe spaces where women can share their unique experiences – failures and successes – when facing challenges in the tech industry.

Adding skills is particularly helpful to get started and increase confidence.

TechFems: As a member of this community, what role do you play in fostering a supportive and inclusive environment for women in tech?

I am in the community as a coach. I enjoy 1:1 interactions where I learn so much from women just entering the field; we have fantastic conversations. I love listening to their experiences. Even though I cannot join events as often as I wish, I’ve greatly enjoyed each event I attended.

Apart from being a coach, I am a teacher assistant for some knowledge sessions. This I enjoy the most as I like hands-on sessions where we learn skills.

Finally, I aim to spread the word with companies that can support the organization with their space for sessions or financial support to provide snacks & refreshments at events. It is easier for coaches already working at companies to ask for help internally. And we can also use our networks to help TechFems connect with those who can support the community.

I started in a men’s world 25 years ago when these communities were unavailable. What would have been different in my career if that support existed? I would have struggled less, for sure.

TechFems: Could you highlight a specific achievement or project within this organization that you are particularly proud of and that has positively impacted women in tech?

I particularly love workshops (e.g., a GitHub actions session) and general “hands-on” sessions like creating your LinkedIn profile or practicing for a coding interview. Adding skills is particularly helpful to get started and increase confidence.

For coaches, the networking sessions have allowed us to connect with many other women and broaden our network. And as a bonus point, we sometimes have our own knowledge sessions, with insightful workshops.

TechFems: How has being a part of this community influenced your professional growth and development in the tech field?

I started in a men’s world 25 years ago when these communities were unavailable. What would have been different in my career if that support existed? I would have struggled less, for sure. Even if I feel privileged – I found my place in tech and have stayed – it’s great to meet other women and help each other thrive. Feeling represented is necessary, and these sessions highlight we are not alone.

TechFems: What advice would you give to other women aspiring to pursue a tech career, and how can organizations like this one contribute to their success?

Come check it for yourself. You’ll learn if a tech career is for you. Don’t be shy with coaches or anyone in the session. Ask anything! It’s a safe space, and all questions are good questions.

Also, nowadays, having an engineering degree is optional. There are many more options to learn and many different paths to take to make a career in tech. Any way that works for you is good.

TechFems: In your view, what initiatives or programs could further enhance the impact of organizations supporting women in tech – like TechFems, and how can members actively contribute to these efforts?

I’d like to see options for mentoring 1:1. Mentoring has a mind-blowing impact, both for mentors and mentees.

And we could also do some hackathons. It helps to realize that we can all be creative in tech and that previous experiences not in tech are also helpful; they add to your new career. Changing careers is not starting from scratch: you bring soft skills, domain knowledge, and many valuable abilities.

TechFems: Is there anything you would like to share?

Keep rocking!

TechFems: Thank you very much for your time and support!

We are excited to announce a significant change to the k6 community forum. Starting on July 25th, at 14:00 UTC, we will merge the k6 community forum with the Grafana community forum.

This migration will create a unified forum for the entire Grafana community, so users no longer need two accounts. It will also provide k6 users access to Grafana Cloud and its free tier, including Grafana Cloud k6.

The k6 and Grafana community forums will be temporarily switched to read-only mode on July 25th at 14:00 UTC. Users cannot log in during the migration process, create new topics, or update existing ones during the merge process.

We expect that the migration process will take 24 hours to complete.

We thank you for your patience during the migration. The unification will provide an enhanced and consolidated Grafana community experience. And it will make it easier for us to operate the forum across the distinct Grafana products.

We encourage you to continue using the community forum to post your k6 questions. Let’s continue to build an even more vital and vibrant community together!

Frequently Asked Questions (FAQ)

1. Where are all the existing k6 topics?

k6 topics in the Grafana forum are under the new “Grafana k6” category under the “Grafana” one. The current k6 categories (OSS Support, Cloud Support, Extensions, etc.) are now subcategories.

Navigate to the “Grafana k6” category and browse or search for your topics.

2. How do I open a k6 topic in the Grafana forum?

Once back in write mode, click “New Topic” to open a k6 topic in the forum.

Select the category “Grafana k6” to ensure the k6 community can find your topic. Add a relevant subcategory (OSS Support, Cloud Support, Extensions, etc.) to improve engagement.

3. How do I log in back to the forum?

To log in after the migration, go to the Grafana Community forum, and:

• If you have a user in the Grafana community forum with the same e-mail you had in the k6 forum, or a Grafana Cloud Account, use the Grafana community login and password. If you forgot your k6 forum password, use the password recovery utility on the login page.
• 

• If you don’t have a user in the Grafana community forum with the same e-mail you had in the k6 forum, you need to:
  • Click the “Log In” button.
  • 
  • Click “Sign Up” for a Grafana Cloud Account.
  • 
  • Create the account with the same e-mail you had in the k6 forum.
  • 
  • Activate the account via the link you’ll receive in the e-mail.
  • 
  • Log in with the Grafana Cloud Account you just created.

4. How can I report issues?

If you cannot log in and password recovery does not work for you, contact community@grafana.com, and we’ll get back to you as soon as possible.

For other non-login-related issues, open a topic under the “Site Feedback” category, and we’ll be happy to assist you.

You can also use the Grafana Community Slack channel #help-grafana-community-forum.

5. What if I have a Grafana forum account with the same e-mail address as my k6 forum account?

In this situation, log in using Grafana’s password, not k6’s.

Upon logging in, you should be able to see all your topics from both Grafana and k6 forums associated with your user.

It is important to note that we don’t migrate most profile data in this scenario: name, username, avatar, etc. Your user keeps the values it had on the Grafana forum.

If you encounter any issues after the migration, contact community@grafana.com. We will do our best to address them as soon as possible.

We appreciate your support and understanding during the migration process. We are looking forward to seeing the unified community grow. And we encourage you to continue using the community forums to engage with and help other k6 community members and the broader Grafana community.

If you have any further questions or concerns, please don’t hesitate to contact us.

1. Know your REST APIs

Knowing Elasticsearch REST APIs is very useful to keep your cluster healthy. Not only can they help you troubleshoot, but prevent issues. If you want something more human-readable, have a look at the CAT APIs.

The first thing to keep the cluster healthy is to keep it in green health. With a simple call to the cluster health API:

GET /_cluster/health

We’ll get an overview of our cluster status.

{
  "cluster_name": "eyeveebee-prod-cluster",
  "status": "red",
  "timed_out": false,
  "number_of_nodes": 30,
  "number_of_data_nodes": 27,
  "active_primary_shards": 15537,
  "active_shards": 26087,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 1,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 209,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 235940,
  "active_shards_percent_as_number": 99.99616762873977
}

In the case above, we have a cluster in red status. We can see we have one unassigned shard, which is causing the cluster to be red. The health of a cluster is that of the worst shard of the worst index. Therefore, at least some index or shard will be in red.

It’s important that we keep our cluster in green health. Kibana alerts can help us here, notifying us when the cluster becomes yellow (missing at least one replica shard) or red (missing at least one primary shard).

To further investigate what index is red, we can use the CAT indices API:

GET _cat/indices?v&s=health:desc,index&h=health,status,index,docs.count,pri,rep

Where we could locate the red index.

health status index                    docs.count pri rep
red    open   eventlogs-000007                      1   1
green  open   .apm-agent-configuration          0   1   1
...

With the CAT shards API we can have look at the shards for the red index ‘eventlogs-000007’:

GET _cat/shards?v&s=state:asc,node,index&h=index,shard,prirep,state,docs,node

Where we would be able to determine that indeed we are missing a primary shard, which is ‘UNASSIGNED’.

index                                                                 shard prirep state         docs node
eventlogs-000007                                                      0     p      UNASSIGNED         
.apm-agent-configuration                                              0     p      STARTED          0 instance-0000000012
...

Finally, we can use the Cluster Allocation explain API to find the reason why.

GET _cluster/allocation/explain
{
  "index": "eventlogs-000007",
  "shard": 0,
  "primary": true
}

We would get an explanation similar to the following, which should allow us to get to the root cause.

{
  "index" : "eventlogs-000007",
  "shard" : 0,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "NODE_LEFT",
    "at" : "2021-12-08T17:00:53.596Z",
    "details" : "node_left [gyv9cseHQyWD-FjLTfSnvA]",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions" : [
    {
      "node_id" : "-PYVqLCLTSKjriA6UZthuw",
      "node_name" : "instance-0000000012",
      "transport_address" : "10.43.1.6:19294",
      "node_attributes" : {
        "xpack.installed" : "true",
        "data" : "hot",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "deciders" : [
        {
          "decider" : "same_shard",
          "decision" : "NO",
          "explanation" : "the node is above the high watermark cluster setting [cluster.routing.allocation.disk.watermark.high=90.0%], having less than the minimum required [90.0%] free space, actual free: [9.13%]"
        }
      ]
    },
    ...

In this case, we ran out of storage on our nodes and shards cannot be assigned. Something we could further confirm using the CAT allocation API.

We can prevent hitting the disk watermarks with Kibana alerts for disk usage threshold. If we keep our nodes around 75% storage, we will have room for growth and keep our cluster healthier. Starting at 85% storage used (default value), we’ll have shard allocation limitations.

We can’t stress enough the importance of planning for data retention. Index Lifecycle Management and the use of Data Tiers are of great help to keep storage in check. Also good to read the documentation on Size your shards, which is also key in keeping a healthy cluster.

Don’t forget to use Elasticsearch snapshots, with snapshot lifecycle-management, to prepare for scenarios where we could lose data if we have no backup. And to set up for high availability.

We also observe in the cluster health above that the number of pending tasks could be a bit high. And we could further look at those again with a call to the CAT pending tasks API:

GET /_cat/pending_tasks?v

To find out what tasks we have pending.

insertOrder timeInQueue priority source 
412717 5.2s NORMAL restore_snapshot[2021.12.17-.ds-logs-2021.12.09-000030-trdkfuewwkjaca] 
412718 2s NORMAL ilm-execute-cluster-state-steps [{"phase":"cold","action":"searchable_snapshot","name":"wait-for-index-color"} => {"phase":"cold","action":"searchable_snapshot","name":"copy-execution-state"}]
...

Or we could also use jq to aggregate the results of the pending cluster tasks API to more easily investigate what are the tasks we have pending.

curl --silent --compressed  'https://localhost:9200/_cluster/pending_tasks' | jq '.tasks[].source' -cMr  | sed -e 's/\[.*//' | sort | uniq -c

Which could give us a better idea of what is causing pending tasks:

  1 restore_snapshot
 17 delete-index
183 ilm-execute-cluster-state-steps
  2 node-join
  4 update task state
  2 update-settings

This is just to showcase that knowing the REST APIs available we can get very helpful information to assess our cluster’s health and adjust our architecture accordingly.

Finally, have a peek at Elastic’s support diagnostics. Those are the REST APIs calls that we use at Elastic Support to help our customers keep their clusters healthy. Or have a look at our blog “Why does Elastic support keep asking for diagnostic files” which explains the underlying reasons why.

2. Take Advantage of the Stack Monitoring & Alerting

The second recommendation is to plan for a separate monitoring cluster when in production. The REST APIs give us current information, but we are missing the historic data. If we send that to a monitoring cluster, it will help us investigate incidents, forecast capacity, etc.

Kibana alerts for the Elastic Stack monitoring will notify us of potential issues.

One example we see a lot in Elastic support, where monitoring comes in handy, is node hot-spotting.

If we have a cluster that is showing high CPU, let’s say, during ingestion, on one or just a few of the data nodes, while the others are idle; and those nodes keep changing. We can use monitoring to confirm our suspicions.

Let’s have a look at the Kibana Stack Monitoring UI for our cluster. Out of 3 nodes, 1 is showing high CPU usage.

We could investigate further by going to the indices tab. We might find, like in this case, that during the window when we see high CPU usage on one node, we have an index ‘log-201998’ that had a very high ingest rate compared to the rest.

If this index has one primary shard, and it’s the only one with a high ingest rate, we could assign 3 primary shards, so the load is balanced between the 3 data instances we have in this example.

For bigger clusters and more than one hot index, the situation might be not so straightforward. We might need to limit the number of shards for those indices that end up on each cluster node. Check on our docs to avoid node hotspots.

Having a monitoring cluster will be of great help.

3. Proactively check Logs

One last recommendation is to proactively review the logs.

We can use Filebeat’s Elasticsearch module to ingest our logs in the monitoring cluster we discussed in the previous section. And even use the stack capabilities to categorize logs to discover anything abnormal and alert us.

One example we see with our customers a lot is wrong mapping data types in the indices.

Depending on how we configure our index mappings, we might be losing documents that come with conflicting types. If we check our cluster logs, we would see those errors and be able to act.

Let’s take the example of a document that sometimes has a numeric value in the source, and sometimes it’s alphanumeric. If we use the default dynamic mappings, and we first ingest this document with a numeric value of a field we’ll call “key”:

POST my-test-index/_doc
{
  "key": 0
}

Elasticsearch will interpret this field as a number, of type long.

GET my-test-index/_mapping/field/key

{
  "my-test-index" : {
    "mappings" : {
      "key" : {
        "full_name" : "key",
        "mapping" : {
          "key" : {
            "type" : "long"
          }
        }
      }
    }
  }
}

If the next document came with, let’s say, with a UUID:

POST my-test-index/_doc
{
  "key": "123e4567-e89b-12d3-a456-426614174000"
}

In Kibana we would see the error.

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse field [key] of type [long] in document with id '4Wtyz30BtaU7QP7QuSQY'. Preview of field's value: '123e4567-e89b-12d3-a456-426614174000'"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse field [key] of type [long] in document with id '4Wtyz30BtaU7QP7QuSQY'. Preview of field's value: '123e4567-e89b-12d3-a456-426614174000'",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "For input string: \"123e4567-e89b-12d3-a456-426614174000\""
    }
  },
  "status" : 400
}

This status code 400 is a non-retriable error, and it means Elasticsearch will not index the document, and clients like Logstash or Agent won’t retry.

If we search for our documents on the index, we only have the first one.

GET my-test-index/_search?filter_path=hits.total,hits.hits._source

{
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "hits" : [
      {
        "_source" : {
          "key" : 0
        }
      }
    ]
  }
}

This that we see in Kibana, would appear in our logs too. It’s very common to find this in Logstash (logs we can also ingest using Filebeat’s Logstash module). It could go unnoticed unless we check our logs.

Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"my-test-index-0000001", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x5662a9f3>], :response=>{"index"=>{"_index"=>"my-test-index-0000001", "_type"=>"_doc", "_id"=>"Qmt6z30BtaU7QP7Q4SXE", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [key] of type [long] in document with id 'Qmt6z30BtaU7QP7Q4SXE'

As a bonus, if we ingest logs, we will be ready to discover slow logs in case we need to troubleshoot any search or ingest slowness.

We’ve recently introduced two additional keyword types, wildcard and constant_keyword. In this post, we’ll try to introduce them briefly.

The wildcard field is optimized to match any part of string values using wildcards or regular expressions. The usual use case is for security when we might search for a pattern in a process or run grep-like queries on log lines that have not been modeled into different fields.

This was introduced in version 7.9. and we’ll demonstrate this with a basic example. We’ll be using Kibana sample data “Sample weblogs” on a Kibana 7.10.0..

We want to get all the different zip files users downloaded from our website. With the current sample data, we could run the following query:

GET kibana_sample_data_logs/_search?filter_path=aggregations.zip-downloads.buckets.key
{
  "size": 0, 
  "_source": "request", 
  "query": {
    "wildcard": {
      "url.keyword": {
        "value": "*downloads*.ZIP",
        "case_insensitive": true
      }
    }
  },
  "aggs": {
    "zip-downloads": {
      "terms": {
        "field": "url.keyword",
        "size": 10
      }
    }
  }
}

Which is using the field url.keyword, already existing in the index kibana_sample_data_logs, to run the query. And would return:

{
  "aggregations" : {
    "zip-downloads" : {
      "buckets" : [
        {
          "key" : "https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.zip"
        },
        {
          "key" : "https://artifacts.elastic.co/downloads/apm-server/apm-server-6.3.2-windows-x86.zip"
        },
        {
          "key" : "https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-windows-x86_64.zip"
        }
      ]
    }
  }
}

Once we have the index kibana_sample_data_logs, we can update its mappings to add to the existing fields url (type text) and url.keyword (type keyword), a third field url.wirldcard of type wildcard.

PUT kibana_sample_data_logs/_mappings
{
  "properties" : {
        "url" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            },
            "wildcard" : {
              "type" : "wildcard"
            }
          }
        }
      }
}

To populate the field, we can run an update:

POST kibana_sample_data_logs/_update_by_query

Now we can execute the same query using the url.wildcard field.

GET kibana_sample_data_logs/_search?filter_path=aggregations.zip-downloads.buckets.key
{
  "size": 0, 
  "_source": "request", 
  "query": {
    "wildcard": {
      "url.wildcard": {
        "value": "*downloads*.ZIP",
        "case_insensitive": true
      }
    }
  },
  "aggs": {
    "zip-downloads": {
      "terms": {
        "field": "url.keyword",
        "size": 10
      }
    }
  }
}

What we’ve done is replace a costly leading wildcard query on a keyword field, with a leading wildcard search on a wildcard field.

How does this help us?

• If we have a field with high cardinality, a keyword is not optimized for leading wildcard searches, and thus, it will have to scan all the registries. Wildcard fields index the whole field value using ngrams and store the entire string. Combining both data structures in the search can speed up the search.
• String lengths have a limit. Lucene has a hard limit of 32k on terms, and Elasticsearch imposes an even lower limit. That means the content is dropped from the index if you reach that size. For example, a stack trace can be lengthy. And in some cases, like security, this can create blind spots that are not acceptable.

What you see here is that the same search we had with the keyword field will work on the new wildcard field. In this simple example, the speed will be similar, as we do not have high cardinality in the URL field values for this small data sample. Though you get how it works.

When to use a wildcard? What’s the trade-off? In this post, we won’t get into more details. To further dig into this, go ahead and check the following resources:

• ElasticON Global 2020 talk: Elasticsearch: Introducing the wildcard field
• Blog post: Find Strings within strings faster with the new Elasticsearch wildcard field

Moving on to the second addition to the keyword family, the constant_keyword field, available since version 7.7.0..

This is a field we can use in cases where we want to speed up filter searches.

Usually, the more documents a filter matches, the more that query will cost. For example, if we send all application logs collected for different dev teams to the same index, we might find that later we usually filter them based on the team. If we know this is a usual filter for our use case, we could decide to ingest data on different indices, based on the team field value on each log line. To make queries faster, as they would hit just indices with matching data for that team.

And we can go one step further. Maybe we do not want to change our client’s logic the way they query. We’d like the same queries to work, but still filter out more effectively the indices that have data that won’t match.

If we create those indices with a constant_keyword field, and set the constant in the mappings, the queries sent to all indices will make use of that field to discard indices that won’t match.

We’ll go over it with a basic example to demonstrate how it works.

We’ll create two indices to hold logs for two different teams, A and B. On each index, we define the field team with a constant value, A or B.

PUT my-logs-team-a
{
  "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "message" : {
          "type" : "text"
        },
        "team" : {
          "type" : "constant_keyword",
          "value": "A"
        }
      }
    }
}

PUT my-logs-team-b
{
  "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "message" : {
          "type" : "text"
        },
        "team" : {
          "type" : "constant_keyword",
          "value": "B"
        }
      }
    }
}

We’ll now ingest a document into each index. We can include the field team value in the document to ingest:

POST my-logs-team-a/_doc/
{
  "@timestamp": "2020-12-04T10:00:13.637Z",
  "message": "239.23.215.100 - - [2018-08-10T13:09:44.504Z] \"GET /apm HTTP/1.1\" 200 8679 \"-\" \"Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24\"",
  "team": "A"
}

Or leave the value for that field out, letting it take the one defined in the mapping:

POST my-logs-team-b/_doc/
{
  "@timestamp": "2020-12-04T10:01:12.654Z",
  "message": "34.52.49.238 - - [2018-08-10T12:23:20.235Z] \"GET /apm HTTP/1.1\" 200 117 \"-\" \"Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24\""
}

We will now be able to run the queries in a more efficient way without selecting the indices that contain the values we want to filter. We can search on my-logs-team-*, and Elasticsearch will do its magic for us:

GET my-logs-team-*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "team": "B"
          }
        }
      ]
    }
  }
}

If we run the query on Kibana’s “Search Profiler”, we can see that when searching for team B, we are executing a match_none query on the team-a index. Thus speeding up the filtering operation.

If we created the indices using a type keyword and ran the same test, we would see how both indices run the same term query, even if one of them will return no results.

Go take the two new keyword types for a spin, and let us know how it goes!

This allows us to do lookups on other Elasticsearch indices and enrich the incoming document before sending it to its own index.

Enrich Policies 4 are what powers this new functionality in Elasticsearch version 7.5., and at this point, we can enrich data based on geolocation or through matching 1 exact values using a term query.

Let’s use an example to demonstrate this new capability.

I’m an avid reader at my local library. They have ingested their catalog in Elasticsearch, and each book is searchable in the index book-catalog with the book title, author, and ISBN (the International Standard Book Number). This is a sample of the catalog books:

POST book-catalog/_doc/
{
    "isbn-10": "0571276547",
    "author": "Paul Auster",
    "title": "The Brooklyn Follies"
}

POST book-catalog/_doc/
{
    "isbn-10": "0571283209",
    "author": "Paul Auster",
    "title": "Winter Journal"
}

POST book-catalog/_doc/
{
    "isbn-10": "2298068968",
    "author": "Marc Levy",
    "title": "Un sentiment plus fort que la peur"
}
    
POST book-catalog/_doc/
{ 
    "isbn-10": "086068511",
    "author": "Maya Angelou",
    "title": "I Know Why The Caged Bird Sings"
}

POST book-catalog/_doc/
{
    "isbn-10": "163286696",
    "author": "James Rhodes",
    "title": "Instrumental"
}

Each time someone borrows a book, the library creates a document in another index, let’s call it book-lending, with the user id, which we will imagine it’s the user’s e-mail, and the book ISBN. For example:

POST book-lending/_doc
{
    "isbn-10": "163286696",
    "user": "mim@mail.net"
}

Later on, they would like to explore what are the user’s favorite authors. Or, whenever a new book is acquired from an author, inform users who have borrowed books from the same author in the past.

What will help my librarians get that information in a single query, involves enriching the book lending data at ingest time in a way that the book-lending document holds not only the ISBN code for a book but also the name of the book and author.

This can be achieved as follows.

Create an Enrich Policy 4 that matches 1 the isbn-10 field from the incoming document to the book-catalog index.

PUT /_enrich/policy/book-catalog
{
  "match": {
    "indices": "book-catalog",
    "match_field": "isbn-10",
    "enrich_fields": ["author", "title"]
  }
}

Execute the previous policy.

PUT /_enrich/policy/book-catalog/_execute 

Policies must be executed before any enrich processor can make use of them. Elasticsearch will create a system index, the enrich index, that the processor will use to enrich incoming documents.Let’s check it out:

GET _cat/indices/.enrich-*?v&s=index&h=index,docs.count,store.size

index                              docs.count store.size
.enrich-book-catalog-1575834789863          5      6.8kb

Enrichment indices are read-only and force merged 2 for fast retrieval. We will have to execute the policy again if new data is ingested into the enrichment index, and a new enrichment index will be created.

Create an ingest pipeline that makes use of that enrich policy. In the example, we are using a matching 1 that will add a field called book-details to the incoming book lending documents, which will hold the book’s author and title based on its isbn-10 code.

PUT _ingest/pipeline/enrich_book_lending
{
  "processors": [
    {
      "enrich": {
        "policy_name": "book-catalog",
        "field": "isbn-10",
        "target_field": "book-details"
      }
    }
  ]
}

Ingest the documents in the book-lending using the pipeline created in the step above.

POST book-lending/_doc?pipeline=enrich_book_lending
{
  "isbn-10": "0571276547",
  "user": "mim@mail.net"
}

POST book-lending/_doc?pipeline=enrich_book_lending
{
  "isbn-10": "2298068968",
  "user": "mim@mail.net"
}

POST book-lending/_doc?pipeline=enrich_book_lending
{
  "isbn-10": "086068511",
  "user": "mim@mail.net"
}

POST book-lending/_doc?pipeline=enrich_book_lending
{
  "isbn-10": "0571283209",
  "user": "mim@mail.net"
}

If we now have a look at the documents ingested, we will see the book information was added under the field book-details:

GET book-lending/_search?filter_path=hits.hits._source
{
    "hits" : {
    "hits" : [
        {
        "_source" : {
            "isbn-10" : "163286696",
            "book-details" : {
            "isbn-10" : "163286696",
            "author" : "James Rhodes"
            "title" : "Instrumental"
            },
            "user" : "mim@mail.net"
        }
        },
…

And thus, my librarians could find out what users like to read “Paul Auster” and send them an e-mail when a new book comes in:

GET book-lending/_search?filter_path=hits.hits._source.user
{
    "query": {
    "bool": {
        "filter": [
            { "term":  { "book-details.author.keyword": "Paul Auster" }}      ]
    }
    },
    "collapse": {
    "field": "user.keyword"
    }
}
__________________
{
    "hits" : {
    "hits" : [
        {
        "_source" : {
            "user" : "mim@mail.net"
        }
        }
    ]
    }
}

Or what is the favorite author for each user:

GET book-lending/_search?filter_path=aggregations.top_author.buckets
{
    "size": 0,
    "query": {
    "bool": {
        "filter": 
            { "term": { "user.keyword": "mim@mail.net" }}
    }
    },
    "aggs": {
    "top_author": {
        "terms": {
        "field": "book-details.author.keyword",
        "size": 1
        }
    }
    }
}
____________________________

{
    "aggregations" : {
    "top_author" : {
        "buckets" : [
        {
            "key" : "Paul Auster",
            "doc_count" : 2
        }
        ]
    }
    }
}

Enjoy reading books and take the enrich processor for a spin!

First, some context

Hot-Warm architectures are often used when we want to get the most out of our hardware. It is particularly useful when we have time-based data, like logs, metrics, and APM data. Most of these setups rely on the fact that this data is read-only (after ingest) and that indices can be time(or size)-based. So they can be easily deleted based on our desired retention period. In this architecture, we categorize Elasticsearch nodes into two types: ‘hot’ and ‘warm’.

Hot nodes hold the most recent data and thus handle all indexing load. Since recent data is usually the most frequently queried, these nodes will be the most powerful in our cluster: fast storage, high memory and CPU. But that extra power gets expensive, so it doesn’t make sense to store older data that isn’t queried as often on a hot node.

On the other hand, warm nodes will be the ones dedicated to long-term storage in a more cost-efficient way. Data on the warm nodes is not as likely to be queried often and data within the cluster will move from hot to warm nodes based on our planned retention (achieved through shard allocation filtering), while still being available online for queries.

Starting with Elastic Stack 6.3, we’ve been building new features to enhance hot-warm architectures and simplify working with time-based data.

Data rollups were first introduced in version 6.3, to save storage. In time-series data, we want fine-grained detail for the most recent data. But it is very unlikely that we will need the same for historical data, where we will typically look at datasets as a whole. And this is where rollups came in, since starting at version 6.5 we can create, manage and visualize rollup data in Kibana.

Shortly after, we added source-only snapshots. These minimal snapshots will provide a significant reduction of snapshot storage, with the tradeoff of having to reindex data if we want to restore and query. This has been available since version 6.5.

In version 6.6, we released two powerful features, Index Lifecycle Management (ILM) and Frozen Indices.

ILM provides the means to automate your indices management over time. It simplifies moving indices from hot to warm, allows deletion when indices are too old, or automates force merging indices down to one segment.

And for the rest of this blog, we’ll talk about frozen indices.

Why freeze an index?

One of the biggest pain points with “old” data is that, regardless of age, indices still have a significant memory footprint. Even if we place them on cold nodes, they still use heap.

A possible solution could be to close the index. If we close an index, it won’t require memory, but we will need to re-open it to run a search. Reopening indices will incur an operational cost and also require the heap it was using before being closed.

On each node, there is a memory (heap) to storage ratio that will limit the amount of storage available per node. It may vary from as low as 1:8 (memory:data) for memory intensive scenarios, to something close to 1:100 for less demanding memory use cases.

This is where frozen indices come in. What if we could have indices that are still open — keeping them searchable — but do not occupy heap? We could add more storage to data nodes that hold frozen indices, and break the 1:100 ratio, understanding the tradeoff that searches might be slower.

When we freeze an index, it becomes read-only and its transient data structures are dropped from memory. In turn, when we run a query on frozen indices, we will have to load the data structures to memory. Searching a frozen index doesn’t have to be slow. Lucene heavily depends on the filesystem cache which might have enough capacity to retain significant portions of your index in memory. In such a case searches are comparable in speed per shard. Yet, a frozen index is still throttled such that only one frozen shard is executing per node at the same time. This aspect might slow down searches compared to unfrozen indices.

How freezing works

Frozen indices are searched through a dedicated, searched-throttled threadpool. This by default uses a single thread, to ensure that frozen indices are loaded into memory one at a time. If concurrent searches are happening, they will get in the queue to add additional protections to prevent nodes from running out of memory.

So, in a hot-warm architecture, we will now be able to transition indices from hot to warm, and then be able to freeze them before archiving or deleting them, allowing us to reduce our hardware requirements.

Before frozen indices, to reduce infrastructure cost we had to snapshot and archive our data, which adds a significant operational cost. We would have to restore data if we needed to search again. Now, we can keep our historical data available for search, without a significant memory overhead. And if we need to write again on an already froze index, we can just unfreeze it.

How to freeze an Elasticsearch index

Frozen indices are easy to implement in your cluster, so let’s get started on how to use the Freeze index API and how to search on frozen indices.

First, we’ll start by creating some sample data on a test index.

POST /sampledata/_doc
{
    "name": "Jane",
    "lastname": "Doe"
}
POST /sampledata/_doc
{
    "name": "John",
    "lastname": "Doe"
}

And then check that our data has been ingested. This should return two hits:

GET /sampledata/_search

As a best practice, before you freeze an index it’s recommended to first run a force_merge. This will ensure that each shard has only a single segment on disk. It will also provide much better compression and simplifies the data structures we will need when running an aggregation or a sorted search request on the frozen index. Running searches on a frozen index with multiple segments can have a significant performance overhead up to multiple orders of magnitude.

POST /sampledata/_forcemerge?max_num_segments=1

The next step is to just invoke a freeze on our index via the Freeze index API endpoint.

POST /sampledata/_freeze

Searching Frozen Indices

Now that it’s frozen, you’ll see that regular searches won’t work. The reason for this is that, to limit memory consumption per node, frozen indices are throttled.  Since we could target a frozen index by mistake, we’ll prevent accidental slowdowns by specifically adding ignore_throttled=false to the request.

GET /sampledata/_search?ignore_throttled=false 
{ 
  "query": { 
    "match": { 
      "name": "jane" 
    } 
  } 
}

Now we can check the status of our new index, by running the following request:

GET _cat/indices/sampledata?v&h=health,status,index,pri,rep,docs.count,store.size

This will return a result similar to the following, with the index status being ‘open’:

health status index      pri rep docs.count store.size
green  open   sampledata   5   1          2     17.8kb

As mentioned above, we must protect the cluster from running out of memory, thus there is a limit in the number of frozen indices we can concurrently load for search on a node. The number of threads in the search-throttled threadpool defaults to 1, with a default queue of 100. This means that if we run more than one request, they will be queued up to a hundred. We can monitor the threadpool status, to check queues and rejections, with the following request:

GET _cat/thread_pool/search_throttled?v&h=node_name,name,active,rejected,queue,completed&s=node_name

Which should return a response similar to:

node_name             name             active rejected queue completed
instance-0000000000   search_throttled      0        0     0        25
instance-0000000001   search_throttled      0        0     0        22
instance-0000000002   search_throttled      0        0     0         0

Frozen indices might be slower, but they can be pre-filtered in a very efficient manner. It is also recommended to set the request parameter pre_filter_shard_size to 1.

GET /sampledata/_search?ignore_throttled=false&pre_filter_shard_size=1
{
 "query": {
   "match": {
     "name": "jane"
   }
 }
}

This will not add a significant overhead to the query and will allow us to take advantage of the usual scenario. For example, when searching on a date range on time-series indices, not all shards will match.

How to write to a frozen Elasticsearch index

What will happen if we try to write on an already frozen index? Let’s go for it and find out.

POST /sampledata/_doc
{
  "name": "Janie",
  "lastname": "Doe"
}

What happened? Frozen indices are read-only, so writing is blocked. We can check these in the index settings:

GET /sampledata/_settings?flat_settings=true

Which will return:

{
 "sampledata" : {
   "settings" : {
     "index.blocks.write" : "true",
     "index.frozen" : "true",
     ....
   }
 }
}

We have to use the Unfreeze index API, invoking the unfreeze endpoint on the index.

POST /sampledata/_unfreeze

And now we’ll be able to create a third document and search for it.

POST /sampledata/_doc
{
 "name": "Janie",
 "lastname": "Doe"
}
GET /sampledata/_search
{
 "query": {
   "match": {
     "name": "janie"
   }
 }
}

Unfreezing should be done only under exceptional situations. And remember to always run a `force_merge` before freezing the index again to ensure optimal performance.

Using frozen indices in Kibana

To begin with, we will need to load some sample data, like the sample flight data.

Click on the “Add” button for Sample flight data.

We should now be able to see the loaded data by clicking the “View data” button. The dashboard will be similar to this one.

Now we can test freezing the index:

POST /kibana_sample_data_flights/_forcemerge?max_num_segments=1
POST /kibana_sample_data_flights/_freeze

And if we go back to our dashboard, we’ll notice that the data has apparently “disappeared”.

We need to tell Kibana to allow searches on frozen indices, which is disabled by default.

Go to Kibana Management, choose Advanced Settings. In the Search section, you will find that “Search in frozen indices” is disabled. Toggle to enable and save the changes.

And the flight’s dashboard will show the data again.

Wrapping up

Frozen Indices are a very powerful tool in hot-warm architectures. They enable a more cost-effective solution for increased retention while retaining online search. I recommend that you test your search latency with your hardware and data, to come with the right sizing and search latency for your frozen indices.

Check out the Elasticsearch documentation to learn more about the Freeze index API. And as always, if you have any questions, reach out on our Discuss forums. Happy freezing!