وقتی از Firebase Remote Config برای استقرار تنظیمات یک برنامه با پایگاه کاربر فعال استفاده میکنید، میخواهید مطمئن شوید که آن را به درستی انجام میدهید. میتوانید از آزمایشهای A/B Testing برای تعیین بهترین موارد زیر استفاده کنید:
- بهترین راه برای پیادهسازی یک ویژگی برای بهینهسازی تجربه کاربری. اغلب، توسعهدهندگان برنامه تا زمانی که رتبه برنامهشان در فروشگاه برنامه کاهش نیابد، متوجه نمیشوند که کاربرانشان از یک ویژگی جدید یا یک تجربه کاربری بهروزرسانیشده خوششان نمیآید. A/B Testing میتواند به سنجش اینکه آیا کاربران شما انواع جدید ویژگیها را دوست دارند یا اینکه برنامه را همانطور که هست ترجیح میدهند، کمک کند. بهعلاوه، نگهداشتن اکثر کاربرانتان در یک گروه پایه تضمین میکند که اکثر پایگاه کاربری شما میتوانند بدون هیچ تغییری در رفتار یا ظاهر برنامه تا زمان پایان آزمایش، به استفاده از آن ادامه دهند.
- بهترین راه برای بهینهسازی تجربه کاربری برای یک هدف تجاری. گاهی اوقات شما در حال اجرای تغییرات محصول برای به حداکثر رساندن معیاری مانند درآمد یا حفظ مشتری هستید. با A/B Testing ، هدف تجاری خود را تعیین میکنید و Firebase تجزیه و تحلیل آماری را انجام میدهد تا مشخص کند که آیا یک متغیر از خط پایه برای هدف انتخابی شما بهتر عمل میکند یا خیر.
برای تست A/B انواع ویژگیها با یک خط پایه، موارد زیر را انجام دهید:
- آزمایش خود را ایجاد کنید.
- آزمایش خود را مدیریت کنید.
یک آزمایش ایجاد کنید
یک آزمایش Remote Config به شما امکان میدهد چندین نوع را روی یک یا چند پارامتر Remote Config ارزیابی کنید.
تأیید کنید که Google Analytics در پروژه شما فعال است تا آزمایش به دادههای Analytics دسترسی داشته باشد.
اگر هنگام ایجاد پروژه خود، Google Analytics فعال نکردهاید، میتوانید آن را در ... فعال کنید.
Firebase تب ادغامها در کنسول فایربیس . در کنسول Firebase ، به DevOps & Engagement > A/B Testing بروید.
روی ایجاد آزمایش کلیک کنید و سپس وقتی از شما خواسته شد سرویسی را که میخواهید با آن آزمایش کنید Remote Config را انتخاب کنید.
در بخش متغیرها ، یک خط مبنا و حداقل یک متغیر برای آزمایش انتخاب کنید. میتوانید یک یا چند پارامتر برای آزمایش اضافه کنید. میتوانید این مرحله را برای اضافه کردن چندین پارامتر به آزمایش خود تکرار کنید.
(اختیاری) برای افزودن بیش از یک نوع به آزمایش خود، روی «افزودن نوع دیگر» کلیک کنید.
یک یا چند پارامتر را برای انواع خاص تغییر دهید. هر پارامتر بدون تغییر برای کاربرانی که در آزمایش گنجانده نشدهاند، یکسان است.
برای مشاهده یا تغییر وزن متغیرها برای آزمایش، وزن متغیرها را باز کنید. به طور پیشفرض، هر متغیر وزن یکسانی دارد. توجه داشته باشید که وزنهای ناهموار ممکن است زمان جمعآوری دادهها را افزایش دهند و وزنها پس از شروع آزمایش قابل تغییر نیستند .
معیارهای هدفگیری برای آزمایش خود را با استفاده از شرایط Remote Config تعریف کنید:
استفاده مجدد از یک شرط موجود: اگر یک شرط موجود در الگوی Remote Config شما از قبل با مخاطب هدف شما مطابقت دارد، آن را از لیست انتخاب کنید.
تأیید ترتیب ارزیابی شرط: مطمئن شوید که شرطهای شما در صفحه شرطها به ترتیب اولویت صحیح سازماندهی شدهاند. از آنجا که Remote Config شرطها را به ترتیب از بالا به پایین ارزیابی میکند، سایر شرطهای با اولویت بالاتر میتوانند مانع از دسترسی تعداد کافی از کاربران به شرط مرتبط با آزمایش شما شوند.
ایجاد یک شرط جدید: اگر هیچ شرط موجودی الزامات هدفگذاری شما را برآورده نمیکند، یا اگر ترجیح میدهید یک شرط موجود را کپی کنید (برای مثال، برای جلوگیری از قفل شدن شرطی که با پارامترهای دیگر مشترک است)، ابتدا با انتخاب برنامهای که از آزمایش شما استفاده میکند، یک شرط جدید ایجاد کنید. اگر پارامتری که آزمایش میکنید از شرط موجود (یا شرط همپوشانی دیگری) نیز استفاده میکند، مطمئن شوید که شرط آزمایش جدید در بالای شرط موجود در برگه شرایط قرار گرفته است (به آن اولویت ارزیابی بالاتری میدهد)؛ در غیر این صورت، کاربران ابتدا با شرط موجود مطابقت داده میشوند و وارد آزمایش نمیشوند.
سپس میتوانید با کلیک کردن و انتخاب یک یا چند گزینه از لیست زیر، زیرمجموعه خاصی از کاربران را هدف قرار دهید:
- نسخه: یک یا چند نسخه از برنامه شما
- شماره ساخت: شماره ساخت (اپل) یا کد نسخه (اندروید) برنامه شما
- پلتفرم: یک یا چند پلتفرم (iOS، اندروید یا وب) برای هدف قرار دادن
- سیستم عامل: کاربران برنامههای وب را بر اساس سیستم عامل و نسخه آنها هدف قرار دهید
- مرورگر: کاربران برنامههای وب را بر اساس مرورگر وب و نسخه مرورگرشان هدف قرار دهید.
- دسته بندی دستگاه: کاربران اپلیکیشن وب را بر اساس اینکه دستگاهشان موبایل است یا غیرموبایل هدف قرار دهید
- زبانها: یک یا چند زبان و منطقهی جغرافیایی که برای انتخاب کاربرانی که ممکن است در آزمایش گنجانده شوند، استفاده میشود.
- کشور/منطقه: یک یا چند کشور یا منطقه برای انتخاب کاربرانی که باید در آزمایش گنجانده شوند
- مخاطبان کاربر: مخاطبان Analytics که برای هدف قرار دادن کاربرانی که ممکن است در آزمایش گنجانده شوند، استفاده میشوند
- ویژگی کاربر: یک یا چند ویژگی کاربر Analytics برای انتخاب کاربرانی که ممکن است در آزمایش گنجانده شوند
- کاربر در درصد تصادفی: درصد تصادفی از کاربران را در یک محدوده صدک تعریف شده هدف قرار دهید.
- بخش وارد شده: کاربرانی را هدف قرار دهید که به بخشهای وارد شده سفارشی آپلود شده در پروژه شما تعلق دارند
- تاریخ/زمان: کاربران را بر اساس یک تاریخ و زمان مشخص هدف قرار دهید
- اولین باز شدن: کاربران را بر اساس اولین باری که برنامه شما را باز کردهاند، هدف قرار دهید.
- شناسه نصب: دستگاههای آزمایشی خاص یا نمونههای کلاینت را با استفاده از شناسههای نصب Firebase (FID) آنها هدف قرار دهید.
- کاربر وجود دارد: همه کاربران را در تمام برنامههای پروژه هدف قرار دهید
- سیگنال سفارشی: کاربران را بر اساس سیگنالهای کلید-مقدار سفارشی سمت کلاینت که در زمان اجرا ارسال میشوند، هدف قرار دهید.
تنظیم میزان مواجهه: درصد پایگاه کاربری برنامه خود را که با معیارهای تعیین شده مطابقت دارد، در قسمت «کاربران هدف» که میخواهید به طور مساوی بین خط پایه و یک یا چند متغیر در آزمایش خود تقسیم کنید، وارد کنید. این میتواند هر درصدی بین 0٪ تا 100٪ باشد. کاربران به طور تصادفی به هر آزمایش، از جمله آزمایشهای تکراری، اختصاص داده میشوند.
به صورت اختیاری، یک رویداد فعالسازی تنظیم کنید تا مطمئن شوید که فقط دادههای کاربرانی که ابتدا یک رویداد Analytics را فعال کردهاند در آزمایش شما شمارش میشوند. توجه داشته باشید که همه کاربرانی که با پارامترهای هدفگیری شما مطابقت دارند، مقادیر آزمایشی Remote Config دریافت خواهند کرد، اما فقط کسانی که یک رویداد فعالسازی را فعال میکنند در نتایج آزمایش شما لحاظ میشوند.
برای اطمینان از یک آزمایش معتبر، مطمئن شوید که رویدادی که انتخاب میکنید پس از فعال شدن مقادیر پیکربندی واکشی شده توسط برنامه شما رخ میدهد. علاوه بر این، رویدادهای زیر قابل استفاده نیستند زیرا همیشه قبل از فعال شدن مقادیر واکشی شده رخ میدهند:
-
app_install -
app_remove -
app_update
رویداد Analytics که به عنوان رویداد فعالسازی انتخاب میکنید، نباید به عنوان معیار اصلی (یا به عنوان یک معیار اضافی) در همان آزمایش نیز استفاده شود. انجام این کار باعث ایجاد خطای اعتبارسنجی در کنسول Firebase شده و از اجرای آزمایش شما جلوگیری میکند.
-
برای اهداف آزمایش، معیار اصلی را برای ردیابی انتخاب کنید و هر معیار اضافی را که میخواهید ردیابی کنید از لیست اضافه کنید. این موارد شامل اهداف از پیش تعیینشده (خریدها، درآمد، حفظ مشتری، کاربران بدون خرابی و غیره)، رویدادهای تبدیل Analytics و سایر رویدادهای Analytics است. پس از اتمام، روی بعدی کلیک کنید.
برای ذخیره آزمایش خود، روی ذخیره کلیک کنید. برای شروع اجرای آزمایش، باید الگو را منتشر کنید.
شما مجاز به انجام حداکثر ۳۰۰ آزمایش در هر پروژه (شامل راهاندازیها) هستید، که میتواند شامل حداکثر ۲۴ آزمایش و راهاندازی در حال اجرا و بقیه آزمایشها به عنوان آزمایشهای تکمیلشده باشد.
آزمایش خود را مدیریت کنید
وقتی با استفاده از Remote Config یک آزمایش ایجاد میکنید، میتوانید آزمایش خود را شروع کنید، در حین اجرا آن را زیر نظر داشته باشید و تعداد کاربرانی که در آزمایش در حال اجرا شرکت میکنند را افزایش دهید.
وقتی آزمایش شما انجام شد، میتوانید تنظیمات مورد استفاده توسط نسخه برنده را یادداشت کنید و سپس آن تنظیمات را برای همه کاربران اعمال کنید. یا میتوانید آزمایش دیگری انجام دهید.
ویرایش یک آزمایش
- در بخش DevOps & Engagement از منوی ناوبری کنسول Firebase ، روی Remote Config کلیک کنید.
- روی برگه تستهای A/B کلیک کنید.
- روی «در حال اجرا» کلیک کنید، روی آزمایشی که میخواهید ویرایش کنید کلیک کنید.
- روی منوی زمینه ( ) کلیک کنید و روی ویرایش آزمایش در حال اجرا کلیک کنید.
- برای تأیید اینکه برنامه شما کاربرانی دارد که میتوانند در آزمایش شما گنجانده شوند، جزئیات را گسترش دهید و در بخش هدفگیری و توزیع، عددی بزرگتر از ۰٪ را بررسی کنید (برای مثال، ۱٪ از کاربران مطابق با معیارها ).
نظارت بر یک آزمایش
وقتی مدتی از اجرای یک آزمایش گذشت، میتوانید پیشرفت آن را بررسی کنید و ببینید نتایج شما برای کاربرانی که تاکنون در آزمایش شما شرکت کردهاند، چگونه به نظر میرسد.
- در بخش DevOps & Engagement از منوی ناوبری کنسول Firebase ، روی Remote Config کلیک کنید.
- روی برگه تستهای A/B کلیک کنید.
روی Running کلیک کنید و سپس روی عنوان آزمایش خود کلیک کنید یا آن را جستجو کنید. در این صفحه، میتوانید آمارهای مختلف مشاهده شده و مدلسازی شده در مورد آزمایش در حال اجرا خود، از جمله موارد زیر را مشاهده کنید:
- درصد اختلاف از حالت پایه : معیاری برای بهبود یک معیار برای یک متغیر معین در مقایسه با حالت پایه. با مقایسه محدوده مقادیر متغیر با محدوده مقادیر حالت پایه محاسبه میشود.
- احتمال غلبه بر خط پایه : احتمال تخمینی که یک متغیر معین، خط پایه را برای معیار انتخاب شده شکست میدهد.
- observed_metric به ازای هر کاربر : بر اساس نتایج آزمایش، این محدوده پیشبینیشدهای است که مقدار متریک در طول زمان در آن قرار خواهد گرفت.
- مجموع observed_metric : مقدار تجمعی مشاهدهشده برای خط پایه یا متغیر. این مقدار برای اندازهگیری میزان عملکرد هر متغیر آزمایشی و محاسبهی بهبود (Improvment)، محدودهی ارزش (Value range )، احتمال بهتر بودن از خط پایه (Probability to Beyond the baseline ) و احتمال بهترین بودن متغیر استفاده میشود. بسته به معیار اندازهگیریشده، این ستون ممکن است با برچسبهای «مدت زمان به ازای هر کاربر»، «درآمد به ازای هر کاربر»، «نرخ ماندگاری» یا «نرخ تبدیل» نامگذاری شود.
پس از اینکه آزمایش شما مدتی اجرا شد (۱۴ روز برای Remote Config )، دادههای این صفحه نشان میدهد که کدام نوع، در صورت وجود، «پیشرو» است. برخی از اندازهگیریها با نمودار میلهای همراه هستند که دادهها را در قالب بصری ارائه میدهد.
یک آزمایش را برای همه کاربران منتشر کنید
بعد از اینکه یک آزمایش به اندازه کافی اجرا شد که شما یک "پیشرو" یا یک نوع برنده برای معیار هدف خود داشتید، میتوانید آزمایش را برای ۱۰۰٪ کاربران منتشر کنید. این به شما امکان میدهد تا یک نوع را برای انتشار برای همه کاربران در آینده انتخاب کنید. حتی اگر آزمایش شما یک برنده مشخص ایجاد نکرده باشد، هنوز میتوانید یک نوع را برای همه کاربران خود منتشر کنید.
- در بخش DevOps & Engagement از منوی ناوبری کنسول Firebase ، روی Remote Config کلیک کنید.
- روی برگه تستهای A/B کلیک کنید.
- روی «تکمیلشده» یا «در حال اجرا» کلیک کنید، روی آزمایشی که میخواهید برای همه کاربران منتشر شود کلیک کنید، روی منوی زمینه ( ) کلیک کنید و گزینه «نوع را منتشر کنید» را انتخاب کنید .
- با انجام موارد زیر، آزمایش خود را برای همه کاربران منتشر کنید:
- برای یک آزمایش Remote Config ، یک نوع را انتخاب کنید تا مشخص شود کدام مقادیر پارامتر Remote Config باید بهروزرسانی شوند. معیارهای هدفگیری که هنگام ایجاد آزمایش تعریف شدهاند، به عنوان یک شرط جدید در الگوی شما اضافه میشوند تا اطمینان حاصل شود که این بهروزرسانی فقط بر کاربرانی که توسط آزمایش هدف قرار گرفتهاند، تأثیر میگذارد. پس از کلیک روی «بررسی» در Remote Config برای بررسی تغییرات، روی «انتشار تغییرات» کلیک کنید تا بهروزرسانی کامل شود.
گسترش یک آزمایش
اگر متوجه شدید که یک آزمایش، کاربران کافی برای A/B Testing و اعلام برتری را جذب نمیکند، میتوانید توزیع آزمایش خود را افزایش دهید تا به درصد بیشتری از پایگاه کاربران برنامه دسترسی پیدا کنید.
- در بخش DevOps & Engagement از منوی ناوبری کنسول Firebase ، روی Remote Config کلیک کنید.
- روی برگه تستهای A/B کلیک کنید.
- آزمایش در حال اجرا را که میخواهید ویرایش کنید، انتخاب کنید.
- در نمای کلی آزمایش ، روی منوی زمینه ( ) کلیک کنید و سپس روی ویرایش آزمایش در حال اجرا کلیک کنید.
- پنجرهی «هدفگذاری» گزینهای را برای افزایش درصد کاربرانی که در آزمایش در حال اجرا هستند نمایش میدهد. عددی بزرگتر از درصد فعلی را انتخاب کنید و روی «انتشار» کلیک کنید. آزمایش به درصد کاربرانی که مشخص کردهاید، منتقل میشود.
تکرار یک آزمایش
- در بخش DevOps & Engagement از منوی ناوبری کنسول Firebase ، روی Remote Config کلیک کنید.
- روی برگه تستهای A/B کلیک کنید.
- آزمایش در حال اجرا یا تکمیلشدهای را که میخواهید متوقف کنید، انتخاب کنید.
- روی «تکمیلشده» یا «در حال اجرا» کلیک کنید، اشارهگر را روی آزمایش خود نگه دارید، روی منوی زمینه ( ) کلیک کنید و سپس روی «تکرار آزمایش» یا «توقف آزمایش» کلیک کنید.
توقف یک آزمایش
- در بخش DevOps & Engagement از منوی ناوبری کنسول Firebase ، روی Remote Config کلیک کنید.
- روی برگه تستهای A/B کلیک کنید.
- آزمایش در حال اجرا یا تکمیلشدهای را که میخواهید متوقف کنید، انتخاب کنید.
- روی «تکمیلشده» یا «در حال اجرا» کلیک کنید، اشارهگر را روی آزمایش خود نگه دارید، روی منوی زمینه ( ) کلیک کنید و سپس روی «توقف آزمایش» کلیک کنید.
شناسایی کلاینت وب و پایداری آزمایش
وقتی کاربری برای اولین بار یک برنامه وب را با استفاده از Firebase A/B Testing در مرورگر اجرا میکند، یک شناسه نصب منحصر به فرد Firebase (FID) ایجاد میشود. این FID به طور مداوم در IndexedDB مرورگر ذخیره میشود تا نمونه برنامه در طول جلسات شناسایی شود.
Firebase A/B Testing از FID برای اختصاص دادن کاربران به انواع آزمایش استفاده میکند و Google Analytics از آن برای تجمیع رویدادها برای اندازهگیری و تحلیل رفتار کاربر در هر نوع استفاده میکند.
از آنجا که FID در IndexedDB ذخیره میشود، Firebase A/B Testing اگر کاربر از مرورگر دیگری یا در یک پنجره ناشناس به برنامه شما دسترسی پیدا کند، یا اگر IndexedDB مرورگر خود را پاک کند، با او به عنوان یک کاربر جدید رفتار میکند. این بدان معناست که یک کاربر ممکن است هنگام استفاده از مرورگرها یا جلسات مرور مختلف، در انواع مختلف آزمایش گنجانده شود.
هدفگیری کاربر
شما میتوانید با استفاده از معیارهای هدفگیری کاربر زیر، کاربرانی را که میخواهید در آزمایش خود بگنجانید، هدف قرار دهید.
انواع قوانین زیر در کنسول Firebase پشتیبانی میشوند. ویژگیهای معادل آن در Remote Config REST API موجود است، همانطور که در مرجع عبارت شرطی توضیح داده شده است.
| نوع قانون | اپراتور(ها) | ارزش(ها) | توجه داشته باشید |
| برنامه | == | از لیست شناسههای برنامه، برنامههای مرتبط با پروژه Firebase خود را انتخاب کنید. | وقتی برنامهای را به Firebase اضافه میکنید، یک شناسه بسته یا نام بسته اندروید وارد میکنید که یک ویژگی را تعریف میکند که به عنوان شناسه برنامه در قوانین Remote Config نمایش داده میشود. از این ویژگی به صورت زیر استفاده کنید:
|
| نسخه برنامه | برای مقادیر رشتهای: دقیقاً مطابقت دارد، شامل، حاوی نیست، شامل عبارت منظم است برای مقادیر عددی: <، <=، =، !=، >، >= | نسخه (های) برنامه خود را برای هدف قرار دادن مشخص کنید. قبل از استفاده از این قانون، باید از یک قانون شناسه برنامه برای انتخاب یک برنامه اندروید/اپل مرتبط با پروژه Firebase خود استفاده کنید. | برای پلتفرمهای اپل: از CFBundleShortVersionString برنامه استفاده کنید. توجه: مطمئن شوید که برنامه اپل شما از SDK پلتفرمهای اپل فایربیس نسخه ۶.۲۴.۰ یا بالاتر استفاده میکند، زیرا CFBundleShortVersionString در نسخههای قبلی ارسال نمیشود (به یادداشتهای انتشار مراجعه کنید). برای اندروید: از versionName برنامه استفاده کنید. نکته: عملگرهای مقایسه عددی ( مقایسههای رشتهای برای این قانون به حروف کوچک و بزرگ حساس هستند. هنگام استفاده از عملگرهای exact matches ، contains ، does not contain یا contains regular expression ، میتوانید چندین مقدار را انتخاب کنید. هنگام استفاده از عملگر عبارت منظم contains ، میتوانید عبارات منظمی با فرمت RE2 ایجاد کنید. عبارت منظم شما میتواند با تمام یا بخشی از رشته نسخه هدف مطابقت داشته باشد. همچنین میتوانید از لنگرهای ^ و $ برای مطابقت با ابتدا، انتها یا کل یک رشته هدف استفاده کنید. |
| شماره ساخت | برای مقادیر رشتهای: دقیقاً مطابقت دارد، شامل، حاوی نیست، عبارت منظم برای مقادیر عددی: =، ≠، >، ≥، <، ≤ | ساخت(های) برنامه خود را برای هدف قرار دادن مشخص کنید. قبل از استفاده از این قانون، باید از یک قانون شناسه برنامه برای انتخاب یک برنامه اپل یا اندروید مرتبط با پروژه Firebase خود استفاده کنید. | این عملگر فقط برای برنامههای اپل و اندروید در دسترس است. این عملگر مربوط به CFBundleVersion برنامه برای اپل و versionCode برای اندروید است. مقایسههای رشتهای برای این قانون به حروف کوچک و بزرگ حساس هستند. هنگام استفاده از عملگرهای exact matches ، contains ، does not contain یا contains عبارت منظم ، میتوانید چندین مقدار را انتخاب کنید. هنگام استفاده از عملگر عبارت منظم contains ، میتوانید عبارات منظمی با فرمت RE2 ایجاد کنید. عبارت منظم شما میتواند با تمام یا بخشی از رشته نسخه هدف مطابقت داشته باشد. همچنین میتوانید از لنگرهای ^ و $ برای مطابقت با ابتدا، انتها یا کل یک رشته هدف استفاده کنید. |
| پلتفرم | == | آیاواس اندروید وب | |
| سیستم عامل | == | سیستم عامل (های) مورد نظر برای هدف گیری را مشخص کنید. قبل از استفاده از این قانون، باید از یک قانون شناسه برنامه برای انتخاب یک برنامه وب مرتبط با پروژه Firebase خود استفاده کنید. | اگر سیستم عامل و نسخه آن با یک مقدار هدف در لیست مشخص شده مطابقت داشته باشند، این قانون برای یک نمونه برنامه وب معین، true ارزیابی میشود. |
| مرورگر | == | مرورگر(های) مورد نظر برای هدف قرار دادن را مشخص کنید. قبل از استفاده از این قانون، باید از یک قانون شناسه برنامه برای انتخاب یک برنامه وب مرتبط با پروژه Firebase خود استفاده کنید. | اگر مرورگر و نسخه آن با یک مقدار هدف در لیست مشخص شده مطابقت داشته باشند، این قانون برای یک نمونه برنامه وب معین، true ارزیابی میشود. |
| دسته بندی دستگاه | هست، نیست | موبایل | این قانون ارزیابی میکند که آیا دستگاهی که به برنامه وب شما دسترسی دارد، موبایل است یا غیرموبایل (دسکتاپ یا کنسول). این نوع قانون فقط برای برنامههای وب در دسترس است. |
| زبانها | در است | یک یا چند زبان را انتخاب کنید. | این قانون برای یک نمونه برنامهی مشخص، در صورتی که آن نمونه برنامه روی دستگاهی نصب شده باشد که از یکی از زبانهای ذکر شده استفاده میکند، true ارزیابی میشود. |
| کشور/منطقه | در است | یک یا چند منطقه یا کشور را انتخاب کنید. | این قانون برای یک نمونه برنامه معین، در صورتی که نمونه در هر یک از مناطق یا کشورهای ذکر شده باشد، true ارزیابی میشود. کد کشور دستگاه با استفاده از آدرس IP دستگاه در درخواست یا کد کشور تعیین شده توسط Firebase Analytics (در صورتی که دادههای Analytics با Firebase به اشتراک گذاشته شود) تعیین میشود. |
| مخاطبان کاربر (بازدیدکنندگان) | شامل حداقل یکی است | یک یا چند مورد از فهرست مخاطبان Google Analytics که برای پروژه خود تنظیم کردهاید را انتخاب کنید. | این قانون برای انتخاب برنامهای مرتبط با پروژه Firebase شما، به یک قانون App ID نیاز دارد. توجه: از آنجا که بسیاری از مخاطبان Analytics توسط رویدادها یا ویژگیهای کاربر تعریف میشوند، که میتواند بر اساس اقدامات کاربران برنامه باشد، ممکن است مدتی طول بکشد تا قانون «کاربر در مخاطب» برای یک نمونه برنامه خاص اعمال شود. این بدان معناست که حتی اگر یک کاربر از نظر فنی واجد شرایط مخاطب باشد، اگر Analytics هنوز کاربر را هنگام اجرای |
| ویژگی کاربر | برای مقادیر رشتهای: شامل، حاوی نیست، دقیقاً مطابقت دارد، شامل عبارت منظم است برای مقادیر عددی: =، ≠، >، ≥، <، ≤ نکته: در نسخه کلاینت، شما فقط میتوانید مقادیر رشتهای را برای ویژگیهای کاربر تنظیم کنید. برای شرایطی که از عملگرهای عددی استفاده میکنند، Remote Config مقدار ویژگی کاربر مربوطه را به یک عدد صحیح/اعشاری تبدیل میکند. | از لیست ویژگیهای کاربر موجود Google Analytics یکی را انتخاب کنید. | برای یادگیری نحوه استفاده از ویژگیهای کاربر برای سفارشیسازی برنامه خود برای بخشهای بسیار خاصی از پایگاه کاربری خود، به Remote Config و ویژگیهای کاربر مراجعه کنید. برای آشنایی بیشتر با ویژگیهای کاربر، به راهنماهای زیر مراجعه کنید: هنگام استفاده از عملگرهای exact matches ، contains ، does not contain یا contains regular expression ، میتوانید چندین مقدار را انتخاب کنید. هنگام استفاده از عملگر عبارت منظم contains ، میتوانید عبارات منظمی با فرمت RE2 ایجاد کنید. عبارت منظم شما میتواند با تمام یا بخشی از رشته نسخه هدف مطابقت داشته باشد. همچنین میتوانید از لنگرهای ^ و $ برای مطابقت با ابتدا، انتها یا کل یک رشته هدف استفاده کنید. توجه: هنگام ایجاد شرایط Remote Config ویژگیهای کاربر که به طور خودکار جمعآوری میشوند، در دسترس نیستند. |
| کاربر به صورت درصد تصادفی | اسلایدر (در کنسول فایربیس. REST API از عملگرهای <= ، > و between استفاده میکند). | ۰-۱۰۰ | از این فیلد برای اعمال تغییر در یک نمونه تصادفی از نمونههای برنامه (با اندازههای نمونه کوچک تا .۰۰۰۱%) استفاده کنید و با استفاده از ویجت اسلایدر، کاربران (نمونههای برنامه) را که به صورت تصادفی انتخاب شدهاند، به گروههایی تقسیم کنید. هر نمونه برنامه، طبق یک سید (seed) تعریف شده در آن پروژه، به طور مداوم به یک عدد صحیح یا کسری تصادفی نگاشت میشود. یک قانون از کلید پیشفرض (که در کنسول Firebase با عنوان Edit seed نشان داده شده است) استفاده میکند، مگر اینکه مقدار seed را تغییر دهید. میتوانید با پاک کردن فیلد Seed ، یک قانون را به استفاده از کلید پیشفرض برگردانید. برای رسیدگی مداوم به نمونههای برنامه مشابه در محدودههای درصد مشخص، از مقدار اولیه یکسانی در شرایط مختلف استفاده کنید. یا با مشخص کردن یک نمونه اولیه جدید، یک گروه جدید از نمونههای برنامه که به صورت تصادفی برای یک محدوده درصد مشخص اختصاص داده شدهاند را انتخاب کنید. برای مثال، برای ایجاد دو شرط مرتبط که هر کدام برای ۵٪ از کاربران یک برنامه که همپوشانی ندارند اعمال میشود، میتوانید یک شرط را طوری پیکربندی کنید که درصدی بین ۰٪ تا ۵٪ را مطابقت دهد و شرط دیگر را طوری پیکربندی کنید که محدودهای بین ۵٪ تا ۱۰٪ را مطابقت دهد. برای اینکه به برخی از کاربران اجازه دهید به طور تصادفی در هر دو گروه ظاهر شوند، از مقادیر اولیه متفاوتی برای قوانین درون هر شرط استفاده کنید. |
| بخش وارداتی | در است | یک یا چند بخش وارد شده را انتخاب کنید. | این قانون مستلزم تنظیم بخشهای سفارشی وارد شده است. |
| تاریخ/زمان | قبل، بعد | تاریخ و زمان مشخص شده، چه در منطقه زمانی دستگاه و چه در یک منطقه زمانی مشخص مانند "(GMT+11) زمان سیدنی". | زمان فعلی را با زمان دریافت دستگاه مقایسه میکند. |
| اولین باز | قبل، بعد | کاربران را بر اساس اولین باری که برنامه شما را باز میکنند، هدف قرار دهید:
| هدفگیری کاربر بر اساس اولین باز شدن، پس از انتخاب یک برنامه اندروید، iOS یا وب، در دسترس است. به SDK های زیر نیاز دارد:
همچنین باید در اولین رویداد باز، Analytics روی کلاینت فعال شده باشد. |
| شناسه نصب | در است | یک یا چند شناسه نصب (تا سقف ۵۰) را برای هدفگیری مشخص کنید. | این قانون برای یک نصب معین، در صورتی که شناسه آن نصب در لیست مقادیر جدا شده با کاما باشد، true ارزیابی میشود.برای آشنایی با نحوهی دریافت شناسههای نصب، به بخش بازیابی شناسههای کلاینت مراجعه کنید. |
| کاربر وجود دارد | (بدون اپراتور) | همه کاربران همه برنامههای موجود در پروژه فعلی را هدف قرار میدهد. | از این قانون شرطی برای تطبیق همه کاربران درون پروژه، صرف نظر از برنامه یا پلتفرم، استفاده کنید. |
| سیگنال سفارشی | برای مقادیر رشتهای: شامل، حاوی نیست، دقیقاً مطابقت دارد، شامل عبارت منظم است برای مقادیر عددی: =، ≠، >، ≥، <، ≤ برای مقادیر نسخه: =، ≠، >، ≥، <، ≤ | مقایسههای رشتهای برای این قانون به حروف کوچک و بزرگ حساس هستند. هنگام استفاده از عملگرهای exact matches، contains، does not contain یا contains عبارت منظم، میتوانید چندین مقدار را انتخاب کنید. هنگام استفاده از عملگر contains عبارت منظم، میتوانید عبارات منظمی با فرمت RE2 ایجاد کنید. عبارت منظم شما میتواند با تمام یا بخشی از رشته نسخه هدف مطابقت داشته باشد. همچنین میتوانید از لنگرهای ^ و $ برای مطابقت با ابتدا، انتها یا کل یک رشته هدف استفاده کنید. انواع داده زیر برای محیطهای کلاینت پشتیبانی میشوند:
عددی که نشان دهنده شماره (شمارههای) نسخهای است که باید مطابقت داده شود (برای مثال، ۲.۱.۰). | برای اطلاعات بیشتر در مورد شرایط سیگنال سفارشی و عبارات شرطی مورد استفاده، به شرایط سیگنال سفارشی و عناصر مورد استفاده برای ایجاد شرایط مراجعه کنید. |
معیارهای A/B Testing
وقتی آزمایش خود را ایجاد میکنید، یک معیار اصلی یا هدف را انتخاب میکنید که برای تعیین نوع برنده استفاده میشود. همچنین باید معیارهای دیگری را پیگیری کنید تا به شما در درک بهتر عملکرد هر نوع آزمایش کمک کند و روندهای مهمی را که ممکن است برای هر نوع متفاوت باشد، مانند حفظ کاربر، پایداری برنامه و درآمد خرید درون برنامه، پیگیری کنید. میتوانید تا پنج معیار غیرهدف را در آزمایش خود پیگیری کنید.
برای مثال، فرض کنید از Remote Config برای راهاندازی دو جریان بازی مختلف در برنامه خود استفاده میکنید و میخواهید خریدهای درونبرنامهای و درآمد تبلیغات را بهینهسازی کنید، اما همچنین میخواهید پایداری و میزان حفظ کاربر هر نوع را پیگیری کنید. در این حالت، میتوانید Estimated total income را به عنوان معیار هدف خود انتخاب کنید زیرا شامل درآمد خرید درونبرنامهای و درآمد تبلیغات میشود و سپس، برای سایر معیارها برای پیگیری ، میتوانید موارد زیر را اضافه کنید:
- برای پیگیری میزان حفظ کاربر روزانه و هفتگی، «حفظ کاربر» (۲-۳ روز) و «حفظ کاربر» (۴-۷ روز) را اضافه کنید.
- برای مقایسه پایداری بین دو جریان بازی، کاربران بدون خرابی (Crash-free users) را اضافه کنید.
- برای مشاهده جزئیات بیشتر هر نوع درآمد، درآمد حاصل از خرید و درآمد تخمینی از تبلیغات را اضافه کنید.
جداول زیر جزئیاتی در مورد نحوه محاسبه معیارهای هدف و سایر معیارها ارائه میدهند.
معیارهای هدف
| متریک | توضیحات |
|---|---|
| کاربران بدون خرابی | درصد کاربرانی که با خطاهایی که توسط Firebase Crashlytics SDK در طول آزمایش شناسایی شدهاند، در برنامه شما مواجه نشدهاند. توجه: Firebase Crashlytics برای برنامههای وب پشتیبانی نمیشود. |
| درآمد تخمینی تبلیغات | درآمد تخمینی از تبلیغات |
| کل درآمد تخمینی | ارزش ترکیبی برای خرید و درآمد تخمینی تبلیغات. |
| درآمد حاصل از خرید | ارزش ترکیبی برای همه رویدادهای purchase و in_app_purchase . |
| ماندگاری (۱ روز) | تعداد کاربرانی که روزانه به اپلیکیشن شما مراجعه میکنند. |
| ماندگاری (۲-۳ روز) | تعداد کاربرانی که ظرف ۲-۳ روز به اپلیکیشن شما بازمیگردند. |
| ماندگاری (۴-۷ روز) | تعداد کاربرانی که ظرف ۴ تا ۷ روز به اپلیکیشن شما بازمیگردند. |
| ماندگاری (۸-۱۴ روز) | تعداد کاربرانی که ظرف ۸ تا ۱۴ روز دوباره به اپلیکیشن شما مراجعه میکنند. |
| ماندگاری (۱۵+ روز) | تعداد کاربرانی که ۱۵ روز یا بیشتر پس از آخرین استفاده از برنامه شما، دوباره به آن مراجعه میکنند. |
| first_open | یک رویداد Analytics که وقتی کاربر پس از نصب یا نصب مجدد برنامه، آن را برای اولین بار باز میکند، فعال میشود. به عنوان بخشی از قیف تبدیل استفاده میشود. |
سایر معیارها
| متریک | توضیحات |
|---|---|
| notification_dismiss | یک رویداد Analytics که هنگام رد شدن اعلان ارسال شده توسط سازنده اعلانها (Notifications composer) فعال میشود (فقط اندروید). |
| دریافت_اطلاعیه | یک رویداد Analytics که زمانی فعال میشود که اعلان ارسال شده توسط سازنده اعلانها (Notifications composer) در حالی که برنامه در پسزمینه است (فقط اندروید) دریافت شود. |
| بهروزرسانی سیستم عامل | یک رویداد Analytics که هنگام بهروزرسانی سیستم عامل دستگاه به نسخه جدید، ردیابی میکند. برای کسب اطلاعات بیشتر، به رویدادهای جمعآوریشده خودکار مراجعه کنید. این معیار برای برنامههای وب پشتیبانی نمیشود. |
| نمای صفحه | یک رویداد Analytics که صفحات مشاهده شده در برنامه شما را ردیابی میکند. برای کسب اطلاعات بیشتر، به Track Screenviews مراجعه کنید. |
| شروع_جلسه | یک رویداد Analytics که تعداد جلسات کاربر را در برنامه شما شمارش میکند. برای کسب اطلاعات بیشتر، به رویدادهای جمعآوریشده خودکار مراجعه کنید. |
خروجی گرفتن از دادههای BigQuery
علاوه بر مشاهده دادههای آزمایش A/B Testing در کنسول Firebase ، میتوانید دادههای آزمایش را در BigQuery بررسی و تجزیه و تحلیل کنید. در حالی که A/B Testing جدول BigQuery جداگانهای ندارد، عضویتهای آزمایش و متغیر در هر رویداد Google Analytics در جداول رویداد Analytics ذخیره میشوند.
ویژگیهای کاربری که حاوی اطلاعات آزمایش هستند، به شکل userProperty.key like "firebase_exp_%" یا userProperty.key = "firebase_exp_01" که در آن 01 شناسه آزمایش است و userProperty.value.string_value شامل اندیس (مبتنی بر صفر) نوع آزمایش است.
شما میتوانید از این ویژگیهای کاربر آزمایش برای استخراج دادههای آزمایش استفاده کنید. این به شما قدرت میدهد تا نتایج آزمایش خود را به روشهای مختلفی برش دهید و نتایج A/B Testing را به طور مستقل تأیید کنید.
برای شروع، موارد زیر را طبق توضیحات این راهنما انجام دهید:
- فعال کردن خروجی BigQuery برای Google Analytics در کنسول فایربیس
- دسترسی به دادههای A/B Testing با استفاده از BigQuery
- کاوش در نمونه سوالات
فعال کردن خروجی BigQuery برای Google Analytics در کنسول فایربیس
اگر از طرح Spark استفاده میکنید، میتوانید از محیط سندباکس BigQuery برای دسترسی رایگان BigQuery استفاده کنید، البته با توجه به محدودیتهای Sandbox . برای اطلاعات بیشتر به بخش قیمتگذاری و محیط سندباکس BigQuery مراجعه کنید.
ابتدا مطمئن شوید که دادههای Analytics خود را به BigQuery منتقل میکنید:
در کنسول Firebase ، به مسیر زیر بروید:
> برگه یکپارچهسازیها . در کارت BigQuery ، روی مدیریت (Manage) کلیک کنید و تأیید کنید که پروژه شما دادههای Analytics را به BigQuery صادر میکند.
اگر روی کارت عبارت «پیوند» نوشته شده باشد، باید خروجی گرفتن را تنظیم کنید (به مرحله بعدی بروید).
اگر نیاز به تنظیم صادرات دارید:
درباره اتصال Firebase به BigQuery نظر بدهید، سپس روی Next کلیک کنید.
در بخش پیکربندی ادغام ، Google Analytics فعال کنید.
یک منطقه را انتخاب کنید و تنظیمات صادرات را انتخاب کنید.
روی پیوند به BigQuery کلیک کنید.
بسته به نحوهی انتخاب شما برای خروجی گرفتن از دادهها، ممکن است تا یک روز طول بکشد تا جداول در دسترس قرار گیرند. برای اطلاعات بیشتر در مورد خروجی گرفتن از دادههای پروژه به BigQuery ، به بخش خروجی گرفتن از دادههای پروژه به BigQuery مراجعه کنید.
دسترسی به دادههای A/B Testing در BigQuery
قبل از جستجوی دادهها برای یک آزمایش خاص، باید برخی یا همه موارد زیر را برای استفاده در جستجوی خود به دست آورید:
- شناسه آزمایش: میتوانید این را از آدرس اینترنتی صفحه مرور کلی آزمایش دریافت کنید. برای مثال، اگر آدرس اینترنتی شما به شکل
https://console.firebase.google.com/project/my_firebase_project/config/experiment/results/25باشد، شناسه آزمایش ۲۵ است. - شناسه ویژگی Google Analytics : این شناسه ۹ رقمی ویژگی Google Analytics شماست. میتوانید آن را در Google Analytics پیدا کنید؛ همچنین در BigQuery وقتی نام پروژه خود را باز میکنید تا نام جدول رویداد Google Analytics شما (
project_name.analytics_000000000.events) نمایش داده میشود، ظاهر میشود. - Experiment date: To compose a faster and more efficient query, it's good practice to limit your queries to the Google Analytics daily event table partitions that contain your experiment data—tables identified with a
YYYYMMDDsuffix. So, if your experiment ran from February 2, 2024 through May 2, 2024, you'd specify a_TABLE_SUFFIX between '20240202' AND '20240502'. For an example, see Select a specific experiment's values . - Event names: Typically, these correspond with your goal metrics that you configured in the experiment. For example,
in_app_purchaseevents,ad_impression, oruser_retentionevents.
After you gather the information you need to generate your query:
- In the Google Cloud console, go to BigQuery .
- Select your project, then select Create SQL query .
- Add your query. For example queries to run, see Explore example queries .
- Click Run .
Query experiment data using the Firebase console's auto-generated query
If you're using the Blaze plan, the Experiment overview page provides a sample query that returns the experiment name, variants, event names, and the number of events for the experiment you're viewing.
To obtain and run the auto-generated query:
- In the Firebase console, go to DevOps & Engagement > A/B Testing .
- Select the A/B Testing experiment you want to query to open the Experiment overview .
- From the Options menu, beneath BigQuery integration , select Query experiment data . This opens your project in BigQuery within the Google Cloud console console and provides a basic query you can use to query your experiment data.
The following example shows a generated query for an experiment with three variants (including the baseline) named "Winter welcome experiment." It returns the active experiment name, variant name, unique event, and event count for each event. Note that the query builder doesn't specify your project name in the table name, as it opens directly within your project.
/*
This query is auto-generated by Firebase A/B Testing for your
experiment "Winter welcome experiment".
It demonstrates how you can get event counts for all Analytics
events logged by each variant of this experiment's population.
*/
SELECT
'Winter welcome experiment' AS experimentName,
CASE userProperty.value.string_value
WHEN '0' THEN 'Baseline'
WHEN '1' THEN 'Welcome message (1)'
WHEN '2' THEN 'Welcome message (2)'
END AS experimentVariant,
event_name AS eventName,
COUNT(*) AS count
FROM
`analytics_000000000.events_*`,
UNNEST(user_properties) AS userProperty
WHERE
(_TABLE_SUFFIX BETWEEN '20240202' AND '20240502')
AND userProperty.key = 'firebase_exp_25'
GROUP BY
experimentVariant, eventName
For additional query examples, proceed to Explore example queries .
Explore example queries
The following sections provide examples of queries you can use to extract A/B Testing experiment data from Google Analytics event tables.
Extract purchase and experiment standard deviation values from all experiments
You can use experiment results data to independently verify Firebase A/B Testing results. The following BigQuery SQL statement extracts experiment variants, the number of unique users in each variant, and sums total revenue from in_app_purchase and ecommerce_purchase events, and standard deviations for all experiments within the time range specified as the _TABLE_SUFFIX begin and end dates. You can use the data you obtain from this query with a statistical significance generator for one-tailed t-tests to verify that the results Firebase provides match your own analysis.
For more information about how A/B Testing calculates inference, see Interpret test results .
/*
This query returns all experiment variants, number of unique users,
the average USD spent per user, and the standard deviation for all
experiments within the date range specified for _TABLE_SUFFIX.
*/
SELECT
experimentNumber,
experimentVariant,
COUNT(*) AS unique_users,
AVG(usd_value) AS usd_value_per_user,
STDDEV(usd_value) AS std_dev
FROM
(
SELECT
userProperty.key AS experimentNumber,
userProperty.value.string_value AS experimentVariant,
user_pseudo_id,
SUM(
CASE
WHEN event_name IN ('in_app_purchase', 'ecommerce_purchase')
THEN event_value_in_usd
ELSE 0
END) AS usd_value
FROM `PROJECT_NAME.analytics_ANALYTICS_ID.events_*`
CROSS JOIN UNNEST(user_properties) AS userProperty
WHERE
userProperty.key LIKE 'firebase_exp_%'
AND event_name IN ('in_app_purchase', 'ecommerce_purchase')
AND (_TABLE_SUFFIX BETWEEN 'YYYYMMDD' AND 'YYYMMDD')
GROUP BY 1, 2, 3
)
GROUP BY 1, 2
ORDER BY 1, 2;
Select a specific experiment's values
The following example query illustrates how to obtain data for a specific experiment in BigQuery . This sample query returns the experiment name, variant names (including Baseline), event names, and event counts.
SELECT
'EXPERIMENT_NAME' AS experimentName,
CASE userProperty.value.string_value
WHEN '0' THEN 'Baseline'
WHEN '1' THEN 'VARIANT_1_NAME'
WHEN '2' THEN 'VARIANT_2_NAME'
END AS experimentVariant,
event_name AS eventName,
COUNT(*) AS count
FROM
`analytics_ANALYTICS_PROPERTY.events_*`,
UNNEST(user_properties) AS userProperty
WHERE
(_TABLE_SUFFIX BETWEEN 'YYYMMDD' AND 'YYYMMDD')
AND userProperty.key = 'firebase_exp_EXPERIMENT_NUMBER'
GROUP BY
experimentVariant, eventName
When you use Firebase Remote Config to deploy settings for an application with an active user base, you want to make sure you get it right. You can use A/B Testing experiments to best determine the following:
- The best way to implement a feature to optimize the user experience. Too often, app developers don't learn that their users dislike a new feature or an updated user experience until their app's rating in the app store declines. A/B Testing can help measure whether your users like new variants of features, or whether they prefer the app as it exists. Plus, keeping most of your users in a baseline group ensures that most of your user base can continue to use your app without experiencing any changes to its behavior or appearance until the experiment has concluded.
- The best way to optimize the user experience for a business goal. Sometimes you're implementing product changes to maximize a metric like revenue or retention. With A/B Testing , you set your business objective, and Firebase performs the statistical analysis to determine if a variant is outperforming the baseline for your selected objective.
To A/B test feature variants with a baseline, do the following:
- Create your experiment.
- Manage your experiment.
یک آزمایش ایجاد کنید
A Remote Config experiment lets you evaluate multiple variants on one or more Remote Config parameters .
Verify that Google Analytics is enabled in your project so that the experiment has access to Analytics data.
If you didn't enable Google Analytics when creating your project, you can enable it in the
Settings > Integrations tab of the Firebase console. In the Firebase console, go to DevOps & Engagement > A/B Testing
Click Create experiment , and then select Remote Config when prompted for the service you want to experiment with.
In the Variants section, choose a baseline and at least one variant for the experiment. You can add one or more parameters to experiment with. You can repeat this step to add multiple parameters to your experiment.
(optional) To add more than one variant to your experiment, click Add another variant .
Change one or more parameters for specific variants. Any unchanged parameters are the same for users not included in the experiment.
Expand Variant Weights to view or change variant weight for the experiment. By default, each variant is weighted equally. Note that uneven weights may increase data collection time and weights cannot be changed after the experiment begins .
Define the Targeting criteria for your experiment using Remote Config conditions:
Reuse an existing condition: If an existing condition in your Remote Config template already matches your target audience, select it from the list.
Verify condition evaluation order: Make sure your conditions on the Conditions page are organized in the correct priority order. Because Remote Config evaluates conditions sequentially from top to bottom, other higher-priority condition(s) can prevent sufficient users from reaching the condition associated with your experiment.
Create a new condition: If no existing condition satisfies your targeting requirements, or if you prefer to duplicate an existing condition (for example, to avoid locking a condition that is shared by other parameters), create a new condition by first choosing the app that uses your experiment. If the parameter you are testing also uses the existing (or another overlapping) condition, make sure the new experiment condition is placed above the existing condition on the Conditions tab (giving it higher evaluation priority); otherwise, users will match the existing condition first and won't flow into the experiment.
You can then target a specific subset of users by clicking and and selecting one or more options from the following list:
- Version: One or more versions of your app
- Build number: The build number (Apple) or version code (Android) of your app
- Platform: One or more platforms (iOS, Android, or Web) to target
- Operating system: Target web app users based on their operating system and version
- Browser: Target web app users based on their web browser and browser version
- Device category: Target web app users based on whether their device is mobile or non-mobile
- Languages: One or more languages and locales used to select users who might be included in the experiment
- Country/Region: One or more countries or regions for selecting users who should be included in the experiment
- User audience: Analytics audiences used to target users who might be included in the experiment
- User property: One or more Analytics user properties for selecting users who might be included in the experiment
- User in random percentage: Target a randomly selected percentage of users within a defined percentile range
- Imported segment: Target users who belong to custom imported segments uploaded to your project
- Date/Time: Target users based on a specified date and time window
- First open: Target users based on the first time they ever opened your app
- Installation ID: Target specific test devices or client instances using their Firebase Installation IDs (FIDs)
- User exists: Target all users across all apps in the project
- Custom signal: Target users based on custom client-side key-value signals passed at runtime
Set the Exposure: Enter the percentage of your app's user base matching the criteria set under Target users that you want to evenly divide between the baseline and one or more variants in your experiment. This can be any percentage between 0% and 100%. Users are randomly assigned to each experiment, including duplicated experiments.
Optionally, set an activation event to ensure that only the data from users who have first triggered some Analytics event are counted in your experiment. Note that all users matching your targeting parameters will receive Remote Config experimental values, but only those who trigger an activation event will be included in your experiment results.
To ensure a valid experiment, make sure that the event you choose occurs after your app activates fetched configuration values. In addition, the following events cannot be used because they always occur before fetched values are activated:
-
app_install -
app_remove -
app_update
The Analytics event you select as the activation event must not also be used as the primary metric (or as an additional metric) in the same experiment. Doing so will trigger a validation error in the Firebase console and prevent your experiment from launching.
-
For the experiment's Goals , select the primary metric to track, and add any additional metrics you want to track from the list. These include built-in objectives (purchases, revenue, retention, crash-free users, etc.), Analytics conversion events, and other Analytics events. When finished, click Next .
Click Save to save your experiment. You must publish the template to start running your experiment.
You are allowed up to 300 experiments per project (including rollouts), which could consist of up to 24 running experiments and rollouts, with the rest as completed experiments.
Manage your experiment
When you create an experiment with Remote Config , you can start your experiment, monitor your experiment while it is running, and increase the number of users included in your running experiment.
When your experiment is done, you can take note of the settings used by the winning variant, and then roll out those settings to all users. Or, you can run another experiment.
Edit an experiment
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
- Click Running , click an experiment that you want to edit.
- Click the context menu ( ) and click Edit running experiment .
- To validate that your app has users who would be included in your experiment, expand the details and check for a number greater than 0% in the Targeting and distribution section (for example, 1% of users matching the criteria ).
Monitor an experiment
Once an experiment has been running for a while, you can check in on its progress and see what your results look like for the users who have participated in your experiment so far.
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
Click Running , and then click, or search for, the title of your experiment. On this page, you can view various observed and modeled statistics about your running experiment, including the following:
- % difference from baseline : A measure of the improvement of a metric for a given variant as compared to the baseline. Calculated by comparing the value range for the variant to the value range for the baseline.
- Probability to beat baseline : The estimated probability that a given variant beats the baseline for the selected metric.
- observed_metric per user : Based on experiment results, this is the predicted range that the metric value will fall into over time.
- Total observed_metric : The observed cumulative value for the baseline or variant. The value is used to measure how well each experiment variant performs, and is used to calculate Improvement , Value range , Probability to beat baseline , and Probability to be the best variant . Depending on the metric being measured, this column may be labeled "Duration per user," "Revenue per user," "Retention rate," or "Conversion rate."
After your experiment has run for a while (14 days for Remote Config ), data on this page indicates which variant, if any, is the "leader." Some measurements are accompanied by a bar chart that presents the data in a visual format.
Roll out an experiment to all users
After an experiment has run long enough that you have a "leader," or winning variant, for your goal metric, you can release the experiment to 100% of users. This lets you select a variant to publish to all users moving forward. Even if your experiment has not created a clear winner, you can still choose to release a variant to all of your users.
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
- Click Completed or Running , click an experiment that you want to release to all users, click the context menu ( ) Roll out variant .
- Roll out your experiment to all users by doing the following:
- For a Remote Config experiment, select a variant to determine which Remote Config parameter values to update. The targeting criteria defined when creating the experiment is added as a new condition in your template, to ensure the rollout only affects users targeted by the experiment. After clicking Review in Remote Config to review the changes, click Publish changes to complete the rollout.
Expand an experiment
If you find that an experiment isn't bringing in enough users for A/B Testing to declare a leader, you can increase distribution of your experiment to reach a larger percentage of the app's user base.
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
- Select the running experiment that you want to edit.
- In the Experiment overview , click the context menu ( ), and then click Edit running experiment .
- The Targeting dialog displays an option to increase the percentage of users who are in the running experiment. Select a number greater than the current percentage and click Publish . The experiment will be pushed out to the percentage of users you have specified.
Duplicate an experiment
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
- Select the running or completed experiment that you want to stop.
- Click Completed or Running , hold the pointer over your experiment, click the context menu ( ), and then click Duplicate experiment or Stop experiment .
Stop an experiment
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
- Select the running or completed experiment that you want to stop.
- Click Completed or Running , hold the pointer over your experiment, click the context menu ( ), and then click Stop experiment .
Web client identification and experiment persistence
When a user launches a web application using Firebase A/B Testing in a browser for the first time, a unique Firebase installation ID ( FID) is generated. This FID is persistently stored in the browser's IndexedDB to identify the app instance across sessions.
Firebase A/B Testing uses the FID to assign users to experiment variants, and Google Analytics uses it for event aggregation to measure and analyze user behavior within each variant.
Because the FID is stored in IndexedDB, Firebase A/B Testing treats a user as a new user if they access your app from a different browser or in an incognito window, or if they clear their browser's IndexedDB. This means that a user might be included in different experiment variants when using different browsers or browsing sessions.
User targeting
You can target the users to include in your experiment using the following user-targeting criteria.
The following rule types are supported in the Firebase console. Equivalent features are available in the Remote Config REST API, as detailed in the conditional expression reference .
| Rule type | اپراتور(ها) | ارزش(ها) | توجه داشته باشید |
| برنامه | == | Select from a list of App IDs for apps associated with your Firebase project. | When you add an app to Firebase, you enter a bundle ID or Android package name that defines an attribute that's exposed as App ID in Remote Config rules. Use this attribute as follows:
|
| نسخه برنامه | For string values: exactly matches, contains, does not contain, contains regular expression For numeric values: <, <=, =, !=, >, >= | Specify the version(s) of your app to target. Before using this rule, you must use an App ID rule to select an Android/Apple app associated with your Firebase project. | For Apple platforms: Use the app's CFBundleShortVersionString . Note: Make sure your Apple app is using Firebase Apple platforms SDK version 6.24.0 or higher, as CFBundleShortVersionString is not being sent in earlier versions (see release notes ). For Android: Use the app's versionName . Note: Numeric comparison operators ( String comparisons for this rule are case-sensitive. When using the exactly matches , contains , does not contain , or contains regular expression operator, you can select multiple values. When using the contains regular expression operator, you can create regular expressions in RE2 format. Your regular expression can match all or part of the target version string. You can also use the ^ and $ anchors to match the beginning, end, or entirety of a target string. |
| شماره ساخت | For string values: exactly matches, contains, does not contain, عبارت منظم For numeric values: =, ≠, >, ≥, <, ≤ | Specify the build(s) of your app to target. Before using this rule, you must use an App ID rule to select an Apple or Android app associated with your Firebase project. | This operator is available for Apple and Android apps only. It corresponds to the app's CFBundleVersion for Apple and versionCode for Android. String comparisons for this rule are case-sensitive. When using the exactly matches , contains , does not contain , or contains regular expression operator, you can select multiple values. When using the contains regular expression operator, you can create regular expressions in RE2 format. Your regular expression can match all or part of the target version string. You can also use the ^ and $ anchors to match the beginning, end, or entirety of a target string. |
| پلتفرم | == | آیاواس اندروید وب | |
| سیستم عامل | == | Specify the operating system(s) to target. Before using this rule, you must use an App ID rule to select a Web app associated with your Firebase project. | This rule evaluates to true for a given Web app instance if the operating system and its version matches a target value in the specified list. |
| مرورگر | == | Specify the browser(s) to target. Before using this rule, you must use an App ID rule to select a Web app associated with your Firebase project. | This rule evaluates to true for a given Web app instance if the browser and its version matches a target value in the specified list. |
| Device category | is, is not | موبایل | This rule evaluates whether the device accessing your web app is mobile or non-mobile (desktop or console). This rule type is only available for web apps. |
| زبانها | در است | Select one or more languages. | This rule evaluates to true for a given app instance if that app instance is installed on a device that uses one of the languages listed. |
| کشور/منطقه | در است | Select one or more regions or countries. | This rule evaluates to true for a given app instance if the instance is in any of the regions or countries listed. The device country code is determined using the device's IP address in the request or the country code determined by Firebase Analytics (if Analytics data is shared with Firebase). |
| User audience(s) | شامل حداقل یکی است | Select one or more from a list of Google Analytics audiences that you have set up for your project. | This rule requires an App ID rule to select an app associated with your Firebase project. Note: Because many Analytics audiences are defined by events or user properties, which can be based on the actions of app users, it may take some time for a User in audience rule to take effect for a given app instance. This means that even if a user technically qualifies for an audience, if Analytics has not yet added the user to the audience when |
| User property | For string values: contains, does not contain, exactly matches, contains regular expression For numeric values: =, ≠, >, ≥, <, ≤ Note: On the client, you can set only string values for user properties. For conditions that use numeric operators, Remote Config converts the value of the corresponding user property into an integer/float. | Select from a list of available Google Analytics user properties. | To learn how you can use user properties to customize your app for very specific segments of your user base, see Remote Config and user properties . To learn more about user properties, see the following guides: When using the exactly matches , contains , does not contain or contains regular expression operator, you can select multiple values. When using the contains regular expression operator, you can create regular expressions in RE2 format. Your regular expression can match all or part of the target version string. You can also use the ^ and $ anchors to match the beginning, end, or entirety of a target string. Note: Automatically collected user properties are not available when creating Remote Config conditions. |
| User in random percentage | Slider (in the Firebase console. The REST API uses the <= , > , and between operators). | ۰-۱۰۰ | Use this field to apply a change to a random sample of app instances (with sample sizes as small as .0001%), using the slider widget to segment randomly-shuffled users (app instances) into groups. Each app instance is persistently mapped to a random whole or fractional number, according to a seed defined in that project. A rule will use the default key (shown as Edit seed in the Firebase console) unless you modify the seed value. You can return a rule to using the default key by clearing the Seed field. To consistently address the same app instances within given percentage ranges, use the same seed value across conditions. Or, select a new randomly-assigned group of app instances for a given percentage range by specifying a new seed. For example, to create two related conditions that each apply to a non-overlapping 5% of an app's users, you could configure one condition to match a percentage between 0% and 5% and configure another condition to match a range between 5% and 10%. To allow some users to randomly appear in both groups, use different seed values for the rules within each condition. |
| Imported segment | در است | Select one or more imported segment. | This rule requires setting up custom imported segments . |
| تاریخ/زمان | قبل، بعد | A specified date and time, either in the device timezone or a specified timezone such as "(GMT+11) Sydney time." | Compares the current time to the device fetch time. |
| First open | قبل، بعد | Target users based on the first time they open your app:
| User targeting by first open is available after you select an Android, iOS, or Web app. Requires the following SDKs:
Analytics must also have been enabled on the client during the first open event. |
| شناسه نصب | در است | Specify one or more Installation IDs (up to 50) to target. | This rule evaluates to true for a given installation if that installation's ID is in the comma-separated list of values.To learn how you can get installation IDs, see Retrieve client identifiers . |
| User exists | (no operator) | Targets all users of all apps within the current project. | Use this condition rule to match all users within the project, regardless of app or platform. |
| سیگنال سفارشی | For string values: contains, does not contain, exactly matches, contains regular expression For numeric values: =, ≠, >, ≥, <, ≤ For version values: =, ≠, >, ≥, <, ≤ | String comparisons for this rule are case-sensitive. When using the exactly matches, contains, does not contain, or contains regular expression operator, you can select multiple values. When using the contains regular expression operator, you can create regular expressions in RE2 format. Your regular expression can match all or part of the target version string. You can also use the ^ and $ anchors to match the beginning, end, or entirety of a target string. The following data types are supported for client environments:
Numeral that represents the version number(s) to match (for example, 2.1.0). | For more information on custom signal conditions and conditional expressions to use, see Custom signal conditions and Elements used to create conditions . |
A/B Testing metrics
When you create your experiment, you choose a primary, or goal metric, that is used to determine the winning variant. You should also track other metrics to help you better understand each experiment variant's performance and track important trends that may differ for each variant, like user retention, app stability and in-app purchase revenue. You can track up to five non-goal metrics in your experiment.
For example, say you're using Remote Config to launch two different game flows in your app and want to optimize for in-app purchases and ad revenue, but you also want to track the stability and user retention of each variant. In this case, you might consider choosing Estimated total revenue as your goal metric because it includes in-app purchase revenue and ad revenue, and then, for Other metrics to track , you might add the following:
- To track your daily and weekly user retention, add Retention (2-3 days) and Retention (4-7 days) .
- To compare stability between the two game flows, add Crash-free users .
- To see more detailed views of each revenue type, add Purchase revenue and Estimated ad revenue .
The following tables provide details on how goal metrics and other metrics are calculated.
Goal metrics
| متریک | توضیحات |
|---|---|
| Crash-free users | The percentage of users who have not encountered errors in your app that were detected by the Firebase Crashlytics SDK during the experiment. Note: Firebase Crashlytics is not supported for web applications. |
| Estimated ad revenue | Estimated ad earnings. |
| Estimated total revenue | Combined value for purchase and estimated ad revenues. |
| Purchase revenue | Combined value for all purchase and in_app_purchase events. |
| Retention (1 day) | The number of users who return to your app on a daily basis. |
| Retention (2-3 days) | The number of users who return to your app within 2-3 days. |
| Retention (4-7 days) | The number of users who return to your app within 4-7 days. |
| Retention (8-14 days) | The number of users who return to your app within 8-14 days. |
| Retention (15+ days) | The number of users who return to your app 15 or more days after they last used it. |
| first_open | An Analytics event that triggers when a user first opens an app after installing or reinstalling it. Used as part of a conversion funnel. |
سایر معیارها
| متریک | توضیحات |
|---|---|
| notification_dismiss | An Analytics event that triggers when a notification sent by the Notifications composer is dismissed (Android only). |
| notification_receive | An Analytics event that triggers when a notification sent by the Notifications composer is received while the app is in the background (Android only). |
| os_update | An Analytics event that tracks when the device operating system is updated to a new version.To learn more, see Automatically collected events . This metric is not supported for web applications. |
| screen_view | An Analytics event that tracks screens viewed within your app. To learn more, see Track Screenviews . |
| session_start | An Analytics event that counts user sessions in your app. To learn more, see Automatically collected events . |
BigQuery data export
In addition to viewing A/B Testing experiment data in the Firebase console, you can inspect and analyze experiment data in BigQuery . While A/B Testing does not have a separate BigQuery table, experiment and variant memberships are stored on every Google Analytics event within the Analytics event tables.
The user properties that contain experiment information are of the form userProperty.key like "firebase_exp_%" or userProperty.key = "firebase_exp_01" where 01 is the experiment ID, and userProperty.value.string_value contains the (zero-based) index of the experiment variant.
You can use these experiment user properties to extract experiment data. This gives you the power to slice your experiment results in many different ways and independently verify the results of A/B Testing .
To get started, complete the following as described in this guide:
- Enable BigQuery export for Google Analytics in the Firebase console
- Access A/B Testing data using BigQuery
- Explore example queries
Enable BigQuery export for Google Analytics in the Firebase console
If you're on the Spark plan, you can use the BigQuery sandbox to access BigQuery at no cost, subject to Sandbox limits . See Pricing and the BigQuery sandbox for more information.
First, make sure that you're exporting your Analytics data to BigQuery :
In the Firebase console, go to the
Settings > Integrations tab . In the BigQuery card, click Manage and verify that your project is exporting Analytics data to BigQuery .
If the card says Link , then you need to set up the export (continue to the next step).
If you need to set up export:
Review About Linking Firebase to BigQuery , then click Next .
In the Configure integration section, enable Google Analytics .
Select a region and choose export settings.
Click Link to BigQuery .
Depending on how you chose to export data, it may take up to a day for the tables to become available. For more information about exporting project data to BigQuery , see Export project data to BigQuery .
Access A/B Testing data in BigQuery
Before querying for data for a specific experiment, you'll want to obtain some or all of the following to use in your query:
- Experiment ID: You can obtain this from the URL of the Experiment overview page. For example, if your URL looks like
https://console.firebase.google.com/project/my_firebase_project/config/experiment/results/25, the experiment ID is 25 . - Google Analytics property ID : This is your 9-digit Google Analytics property ID. You can find this within Google Analytics ; it also appears in BigQuery when you expand your project name to show the name of your Google Analytics event table (
project_name.analytics_000000000.events). - Experiment date: To compose a faster and more efficient query, it's good practice to limit your queries to the Google Analytics daily event table partitions that contain your experiment data—tables identified with a
YYYYMMDDsuffix. So, if your experiment ran from February 2, 2024 through May 2, 2024, you'd specify a_TABLE_SUFFIX between '20240202' AND '20240502'. For an example, see Select a specific experiment's values . - Event names: Typically, these correspond with your goal metrics that you configured in the experiment. For example,
in_app_purchaseevents,ad_impression, oruser_retentionevents.
After you gather the information you need to generate your query:
- In the Google Cloud console, go to BigQuery .
- Select your project, then select Create SQL query .
- Add your query. For example queries to run, see Explore example queries .
- Click Run .
Query experiment data using the Firebase console's auto-generated query
If you're using the Blaze plan, the Experiment overview page provides a sample query that returns the experiment name, variants, event names, and the number of events for the experiment you're viewing.
To obtain and run the auto-generated query:
- In the Firebase console, go to DevOps & Engagement > A/B Testing .
- Select the A/B Testing experiment you want to query to open the Experiment overview .
- From the Options menu, beneath BigQuery integration , select Query experiment data . This opens your project in BigQuery within the Google Cloud console console and provides a basic query you can use to query your experiment data.
The following example shows a generated query for an experiment with three variants (including the baseline) named "Winter welcome experiment." It returns the active experiment name, variant name, unique event, and event count for each event. Note that the query builder doesn't specify your project name in the table name, as it opens directly within your project.
/*
This query is auto-generated by Firebase A/B Testing for your
experiment "Winter welcome experiment".
It demonstrates how you can get event counts for all Analytics
events logged by each variant of this experiment's population.
*/
SELECT
'Winter welcome experiment' AS experimentName,
CASE userProperty.value.string_value
WHEN '0' THEN 'Baseline'
WHEN '1' THEN 'Welcome message (1)'
WHEN '2' THEN 'Welcome message (2)'
END AS experimentVariant,
event_name AS eventName,
COUNT(*) AS count
FROM
`analytics_000000000.events_*`,
UNNEST(user_properties) AS userProperty
WHERE
(_TABLE_SUFFIX BETWEEN '20240202' AND '20240502')
AND userProperty.key = 'firebase_exp_25'
GROUP BY
experimentVariant, eventName
For additional query examples, proceed to Explore example queries .
Explore example queries
The following sections provide examples of queries you can use to extract A/B Testing experiment data from Google Analytics event tables.
Extract purchase and experiment standard deviation values from all experiments
You can use experiment results data to independently verify Firebase A/B Testing results. The following BigQuery SQL statement extracts experiment variants, the number of unique users in each variant, and sums total revenue from in_app_purchase and ecommerce_purchase events, and standard deviations for all experiments within the time range specified as the _TABLE_SUFFIX begin and end dates. You can use the data you obtain from this query with a statistical significance generator for one-tailed t-tests to verify that the results Firebase provides match your own analysis.
For more information about how A/B Testing calculates inference, see Interpret test results .
/*
This query returns all experiment variants, number of unique users,
the average USD spent per user, and the standard deviation for all
experiments within the date range specified for _TABLE_SUFFIX.
*/
SELECT
experimentNumber,
experimentVariant,
COUNT(*) AS unique_users,
AVG(usd_value) AS usd_value_per_user,
STDDEV(usd_value) AS std_dev
FROM
(
SELECT
userProperty.key AS experimentNumber,
userProperty.value.string_value AS experimentVariant,
user_pseudo_id,
SUM(
CASE
WHEN event_name IN ('in_app_purchase', 'ecommerce_purchase')
THEN event_value_in_usd
ELSE 0
END) AS usd_value
FROM `PROJECT_NAME.analytics_ANALYTICS_ID.events_*`
CROSS JOIN UNNEST(user_properties) AS userProperty
WHERE
userProperty.key LIKE 'firebase_exp_%'
AND event_name IN ('in_app_purchase', 'ecommerce_purchase')
AND (_TABLE_SUFFIX BETWEEN 'YYYYMMDD' AND 'YYYMMDD')
GROUP BY 1, 2, 3
)
GROUP BY 1, 2
ORDER BY 1, 2;
Select a specific experiment's values
The following example query illustrates how to obtain data for a specific experiment in BigQuery . This sample query returns the experiment name, variant names (including Baseline), event names, and event counts.
SELECT
'EXPERIMENT_NAME' AS experimentName,
CASE userProperty.value.string_value
WHEN '0' THEN 'Baseline'
WHEN '1' THEN 'VARIANT_1_NAME'
WHEN '2' THEN 'VARIANT_2_NAME'
END AS experimentVariant,
event_name AS eventName,
COUNT(*) AS count
FROM
`analytics_ANALYTICS_PROPERTY.events_*`,
UNNEST(user_properties) AS userProperty
WHERE
(_TABLE_SUFFIX BETWEEN 'YYYMMDD' AND 'YYYMMDD')
AND userProperty.key = 'firebase_exp_EXPERIMENT_NUMBER'
GROUP BY
experimentVariant, eventName
When you use Firebase Remote Config to deploy settings for an application with an active user base, you want to make sure you get it right. You can use A/B Testing experiments to best determine the following:
- The best way to implement a feature to optimize the user experience. Too often, app developers don't learn that their users dislike a new feature or an updated user experience until their app's rating in the app store declines. A/B Testing can help measure whether your users like new variants of features, or whether they prefer the app as it exists. Plus, keeping most of your users in a baseline group ensures that most of your user base can continue to use your app without experiencing any changes to its behavior or appearance until the experiment has concluded.
- The best way to optimize the user experience for a business goal. Sometimes you're implementing product changes to maximize a metric like revenue or retention. With A/B Testing , you set your business objective, and Firebase performs the statistical analysis to determine if a variant is outperforming the baseline for your selected objective.
To A/B test feature variants with a baseline, do the following:
- Create your experiment.
- Manage your experiment.
یک آزمایش ایجاد کنید
A Remote Config experiment lets you evaluate multiple variants on one or more Remote Config parameters .
Verify that Google Analytics is enabled in your project so that the experiment has access to Analytics data.
If you didn't enable Google Analytics when creating your project, you can enable it in the
Settings > Integrations tab of the Firebase console. In the Firebase console, go to DevOps & Engagement > A/B Testing
Click Create experiment , and then select Remote Config when prompted for the service you want to experiment with.
In the Variants section, choose a baseline and at least one variant for the experiment. You can add one or more parameters to experiment with. You can repeat this step to add multiple parameters to your experiment.
(optional) To add more than one variant to your experiment, click Add another variant .
Change one or more parameters for specific variants. Any unchanged parameters are the same for users not included in the experiment.
Expand Variant Weights to view or change variant weight for the experiment. By default, each variant is weighted equally. Note that uneven weights may increase data collection time and weights cannot be changed after the experiment begins .
Define the Targeting criteria for your experiment using Remote Config conditions:
Reuse an existing condition: If an existing condition in your Remote Config template already matches your target audience, select it from the list.
Verify condition evaluation order: Make sure your conditions on the Conditions page are organized in the correct priority order. Because Remote Config evaluates conditions sequentially from top to bottom, other higher-priority condition(s) can prevent sufficient users from reaching the condition associated with your experiment.
Create a new condition: If no existing condition satisfies your targeting requirements, or if you prefer to duplicate an existing condition (for example, to avoid locking a condition that is shared by other parameters), create a new condition by first choosing the app that uses your experiment. If the parameter you are testing also uses the existing (or another overlapping) condition, make sure the new experiment condition is placed above the existing condition on the Conditions tab (giving it higher evaluation priority); otherwise, users will match the existing condition first and won't flow into the experiment.
You can then target a specific subset of users by clicking and and selecting one or more options from the following list:
- Version: One or more versions of your app
- Build number: The build number (Apple) or version code (Android) of your app
- Platform: One or more platforms (iOS, Android, or Web) to target
- Operating system: Target web app users based on their operating system and version
- Browser: Target web app users based on their web browser and browser version
- Device category: Target web app users based on whether their device is mobile or non-mobile
- Languages: One or more languages and locales used to select users who might be included in the experiment
- Country/Region: One or more countries or regions for selecting users who should be included in the experiment
- User audience: Analytics audiences used to target users who might be included in the experiment
- User property: One or more Analytics user properties for selecting users who might be included in the experiment
- User in random percentage: Target a randomly selected percentage of users within a defined percentile range
- Imported segment: Target users who belong to custom imported segments uploaded to your project
- Date/Time: Target users based on a specified date and time window
- First open: Target users based on the first time they ever opened your app
- Installation ID: Target specific test devices or client instances using their Firebase Installation IDs (FIDs)
- User exists: Target all users across all apps in the project
- Custom signal: Target users based on custom client-side key-value signals passed at runtime
Set the Exposure: Enter the percentage of your app's user base matching the criteria set under Target users that you want to evenly divide between the baseline and one or more variants in your experiment. This can be any percentage between 0% and 100%. Users are randomly assigned to each experiment, including duplicated experiments.
Optionally, set an activation event to ensure that only the data from users who have first triggered some Analytics event are counted in your experiment. Note that all users matching your targeting parameters will receive Remote Config experimental values, but only those who trigger an activation event will be included in your experiment results.
To ensure a valid experiment, make sure that the event you choose occurs after your app activates fetched configuration values. In addition, the following events cannot be used because they always occur before fetched values are activated:
-
app_install -
app_remove -
app_update
The Analytics event you select as the activation event must not also be used as the primary metric (or as an additional metric) in the same experiment. Doing so will trigger a validation error in the Firebase console and prevent your experiment from launching.
-
For the experiment's Goals , select the primary metric to track, and add any additional metrics you want to track from the list. These include built-in objectives (purchases, revenue, retention, crash-free users, etc.), Analytics conversion events, and other Analytics events. When finished, click Next .
Click Save to save your experiment. You must publish the template to start running your experiment.
You are allowed up to 300 experiments per project (including rollouts), which could consist of up to 24 running experiments and rollouts, with the rest as completed experiments.
Manage your experiment
When you create an experiment with Remote Config , you can start your experiment, monitor your experiment while it is running, and increase the number of users included in your running experiment.
When your experiment is done, you can take note of the settings used by the winning variant, and then roll out those settings to all users. Or, you can run another experiment.
Edit an experiment
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
- Click Running , click an experiment that you want to edit.
- Click the context menu ( ) and click Edit running experiment .
- To validate that your app has users who would be included in your experiment, expand the details and check for a number greater than 0% in the Targeting and distribution section (for example, 1% of users matching the criteria ).
Monitor an experiment
Once an experiment has been running for a while, you can check in on its progress and see what your results look like for the users who have participated in your experiment so far.
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
Click Running , and then click, or search for, the title of your experiment. On this page, you can view various observed and modeled statistics about your running experiment, including the following:
- % difference from baseline : A measure of the improvement of a metric for a given variant as compared to the baseline. Calculated by comparing the value range for the variant to the value range for the baseline.
- Probability to beat baseline : The estimated probability that a given variant beats the baseline for the selected metric.
- observed_metric per user : Based on experiment results, this is the predicted range that the metric value will fall into over time.
- Total observed_metric : The observed cumulative value for the baseline or variant. The value is used to measure how well each experiment variant performs, and is used to calculate Improvement , Value range , Probability to beat baseline , and Probability to be the best variant . Depending on the metric being measured, this column may be labeled "Duration per user," "Revenue per user," "Retention rate," or "Conversion rate."
After your experiment has run for a while (14 days for Remote Config ), data on this page indicates which variant, if any, is the "leader." Some measurements are accompanied by a bar chart that presents the data in a visual format.
Roll out an experiment to all users
After an experiment has run long enough that you have a "leader," or winning variant, for your goal metric, you can release the experiment to 100% of users. This lets you select a variant to publish to all users moving forward. Even if your experiment has not created a clear winner, you can still choose to release a variant to all of your users.
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
- Click Completed or Running , click an experiment that you want to release to all users, click the context menu ( ) Roll out variant .
- Roll out your experiment to all users by doing the following:
- For a Remote Config experiment, select a variant to determine which Remote Config parameter values to update. The targeting criteria defined when creating the experiment is added as a new condition in your template, to ensure the rollout only affects users targeted by the experiment. After clicking Review in Remote Config to review the changes, click Publish changes to complete the rollout.
Expand an experiment
If you find that an experiment isn't bringing in enough users for A/B Testing to declare a leader, you can increase distribution of your experiment to reach a larger percentage of the app's user base.
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
- Select the running experiment that you want to edit.
- In the Experiment overview , click the context menu ( ), and then click Edit running experiment .
- The Targeting dialog displays an option to increase the percentage of users who are in the running experiment. Select a number greater than the current percentage and click Publish . The experiment will be pushed out to the percentage of users you have specified.
Duplicate an experiment
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
- Select the running or completed experiment that you want to stop.
- Click Completed or Running , hold the pointer over your experiment, click the context menu ( ), and then click Duplicate experiment or Stop experiment .
Stop an experiment
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
- Select the running or completed experiment that you want to stop.
- Click Completed or Running , hold the pointer over your experiment, click the context menu ( ), and then click Stop experiment .
Web client identification and experiment persistence
When a user launches a web application using Firebase A/B Testing in a browser for the first time, a unique Firebase installation ID ( FID) is generated. This FID is persistently stored in the browser's IndexedDB to identify the app instance across sessions.
Firebase A/B Testing uses the FID to assign users to experiment variants, and Google Analytics uses it for event aggregation to measure and analyze user behavior within each variant.
Because the FID is stored in IndexedDB, Firebase A/B Testing treats a user as a new user if they access your app from a different browser or in an incognito window, or if they clear their browser's IndexedDB. This means that a user might be included in different experiment variants when using different browsers or browsing sessions.
User targeting
You can target the users to include in your experiment using the following user-targeting criteria.
The following rule types are supported in the Firebase console. Equivalent features are available in the Remote Config REST API, as detailed in the conditional expression reference .
| Rule type | اپراتور(ها) | ارزش(ها) | توجه داشته باشید |
| برنامه | == | Select from a list of App IDs for apps associated with your Firebase project. | When you add an app to Firebase, you enter a bundle ID or Android package name that defines an attribute that's exposed as App ID in Remote Config rules. Use this attribute as follows:
|
| نسخه برنامه | For string values: exactly matches, contains, does not contain, contains regular expression For numeric values: <, <=, =, !=, >, >= | Specify the version(s) of your app to target. Before using this rule, you must use an App ID rule to select an Android/Apple app associated with your Firebase project. | For Apple platforms: Use the app's CFBundleShortVersionString . Note: Make sure your Apple app is using Firebase Apple platforms SDK version 6.24.0 or higher, as CFBundleShortVersionString is not being sent in earlier versions (see release notes ). For Android: Use the app's versionName . Note: Numeric comparison operators ( String comparisons for this rule are case-sensitive. When using the exactly matches , contains , does not contain , or contains regular expression operator, you can select multiple values. When using the contains regular expression operator, you can create regular expressions in RE2 format. Your regular expression can match all or part of the target version string. You can also use the ^ and $ anchors to match the beginning, end, or entirety of a target string. |
| شماره ساخت | For string values: exactly matches, contains, does not contain, عبارت منظم For numeric values: =, ≠, >, ≥, <, ≤ | Specify the build(s) of your app to target. Before using this rule, you must use an App ID rule to select an Apple or Android app associated with your Firebase project. | This operator is available for Apple and Android apps only. It corresponds to the app's CFBundleVersion for Apple and versionCode for Android. String comparisons for this rule are case-sensitive. When using the exactly matches , contains , does not contain , or contains regular expression operator, you can select multiple values. When using the contains regular expression operator, you can create regular expressions in RE2 format. Your regular expression can match all or part of the target version string. You can also use the ^ and $ anchors to match the beginning, end, or entirety of a target string. |
| پلتفرم | == | آیاواس اندروید وب | |
| سیستم عامل | == | Specify the operating system(s) to target. Before using this rule, you must use an App ID rule to select a Web app associated with your Firebase project. | This rule evaluates to true for a given Web app instance if the operating system and its version matches a target value in the specified list. |
| مرورگر | == | Specify the browser(s) to target. Before using this rule, you must use an App ID rule to select a Web app associated with your Firebase project. | This rule evaluates to true for a given Web app instance if the browser and its version matches a target value in the specified list. |
| Device category | is, is not | موبایل | This rule evaluates whether the device accessing your web app is mobile or non-mobile (desktop or console). This rule type is only available for web apps. |
| زبانها | در است | Select one or more languages. | This rule evaluates to true for a given app instance if that app instance is installed on a device that uses one of the languages listed. |
| کشور/منطقه | در است | Select one or more regions or countries. | This rule evaluates to true for a given app instance if the instance is in any of the regions or countries listed. The device country code is determined using the device's IP address in the request or the country code determined by Firebase Analytics (if Analytics data is shared with Firebase). |
| User audience(s) | شامل حداقل یکی است | Select one or more from a list of Google Analytics audiences that you have set up for your project. | This rule requires an App ID rule to select an app associated with your Firebase project. Note: Because many Analytics audiences are defined by events or user properties, which can be based on the actions of app users, it may take some time for a User in audience rule to take effect for a given app instance. This means that even if a user technically qualifies for an audience, if Analytics has not yet added the user to the audience when |
| User property | For string values: contains, does not contain, exactly matches, contains regular expression For numeric values: =, ≠, >, ≥, <, ≤ Note: On the client, you can set only string values for user properties. For conditions that use numeric operators, Remote Config converts the value of the corresponding user property into an integer/float. | Select from a list of available Google Analytics user properties. | To learn how you can use user properties to customize your app for very specific segments of your user base, see Remote Config and user properties . To learn more about user properties, see the following guides: When using the exactly matches , contains , does not contain or contains regular expression operator, you can select multiple values. When using the contains regular expression operator, you can create regular expressions in RE2 format. Your regular expression can match all or part of the target version string. You can also use the ^ and $ anchors to match the beginning, end, or entirety of a target string. Note: Automatically collected user properties are not available when creating Remote Config conditions. |
| User in random percentage | Slider (in the Firebase console. The REST API uses the <= , > , and between operators). | ۰-۱۰۰ | Use this field to apply a change to a random sample of app instances (with sample sizes as small as .0001%), using the slider widget to segment randomly-shuffled users (app instances) into groups. Each app instance is persistently mapped to a random whole or fractional number, according to a seed defined in that project. A rule will use the default key (shown as Edit seed in the Firebase console) unless you modify the seed value. You can return a rule to using the default key by clearing the Seed field. To consistently address the same app instances within given percentage ranges, use the same seed value across conditions. Or, select a new randomly-assigned group of app instances for a given percentage range by specifying a new seed. For example, to create two related conditions that each apply to a non-overlapping 5% of an app's users, you could configure one condition to match a percentage between 0% and 5% and configure another condition to match a range between 5% and 10%. To allow some users to randomly appear in both groups, use different seed values for the rules within each condition. |
| Imported segment | در است | Select one or more imported segment. | This rule requires setting up custom imported segments . |
| تاریخ/زمان | قبل، بعد | A specified date and time, either in the device timezone or a specified timezone such as "(GMT+11) Sydney time." | Compares the current time to the device fetch time. |
| First open | قبل، بعد | Target users based on the first time they open your app:
| User targeting by first open is available after you select an Android, iOS, or Web app. Requires the following SDKs:
Analytics must also have been enabled on the client during the first open event. |
| شناسه نصب | در است | Specify one or more Installation IDs (up to 50) to target. | This rule evaluates to true for a given installation if that installation's ID is in the comma-separated list of values.To learn how you can get installation IDs, see Retrieve client identifiers . |
| User exists | (no operator) | Targets all users of all apps within the current project. | Use this condition rule to match all users within the project, regardless of app or platform. |
| سیگنال سفارشی | For string values: contains, does not contain, exactly matches, contains regular expression For numeric values: =, ≠, >, ≥, <, ≤ For version values: =, ≠, >, ≥, <, ≤ | String comparisons for this rule are case-sensitive. When using the exactly matches, contains, does not contain, or contains regular expression operator, you can select multiple values. When using the contains regular expression operator, you can create regular expressions in RE2 format. Your regular expression can match all or part of the target version string. You can also use the ^ and $ anchors to match the beginning, end, or entirety of a target string. The following data types are supported for client environments:
Numeral that represents the version number(s) to match (for example, 2.1.0). | For more information on custom signal conditions and conditional expressions to use, see Custom signal conditions and Elements used to create conditions . |
A/B Testing metrics
When you create your experiment, you choose a primary, or goal metric, that is used to determine the winning variant. You should also track other metrics to help you better understand each experiment variant's performance and track important trends that may differ for each variant, like user retention, app stability and in-app purchase revenue. You can track up to five non-goal metrics in your experiment.
For example, say you're using Remote Config to launch two different game flows in your app and want to optimize for in-app purchases and ad revenue, but you also want to track the stability and user retention of each variant. In this case, you might consider choosing Estimated total revenue as your goal metric because it includes in-app purchase revenue and ad revenue, and then, for Other metrics to track , you might add the following:
- To track your daily and weekly user retention, add Retention (2-3 days) and Retention (4-7 days) .
- To compare stability between the two game flows, add Crash-free users .
- To see more detailed views of each revenue type, add Purchase revenue and Estimated ad revenue .
The following tables provide details on how goal metrics and other metrics are calculated.
Goal metrics
| متریک | توضیحات |
|---|---|
| Crash-free users | The percentage of users who have not encountered errors in your app that were detected by the Firebase Crashlytics SDK during the experiment. Note: Firebase Crashlytics is not supported for web applications. |
| Estimated ad revenue | Estimated ad earnings. |
| Estimated total revenue | Combined value for purchase and estimated ad revenues. |
| Purchase revenue | Combined value for all purchase and in_app_purchase events. |
| Retention (1 day) | The number of users who return to your app on a daily basis. |
| Retention (2-3 days) | The number of users who return to your app within 2-3 days. |
| Retention (4-7 days) | The number of users who return to your app within 4-7 days. |
| Retention (8-14 days) | The number of users who return to your app within 8-14 days. |
| Retention (15+ days) | The number of users who return to your app 15 or more days after they last used it. |
| first_open | An Analytics event that triggers when a user first opens an app after installing or reinstalling it. Used as part of a conversion funnel. |
سایر معیارها
| متریک | توضیحات |
|---|---|
| notification_dismiss | An Analytics event that triggers when a notification sent by the Notifications composer is dismissed (Android only). |
| notification_receive | An Analytics event that triggers when a notification sent by the Notifications composer is received while the app is in the background (Android only). |
| os_update | An Analytics event that tracks when the device operating system is updated to a new version.To learn more, see Automatically collected events . This metric is not supported for web applications. |
| screen_view | An Analytics event that tracks screens viewed within your app. To learn more, see Track Screenviews . |
| session_start | An Analytics event that counts user sessions in your app. To learn more, see Automatically collected events . |
BigQuery data export
In addition to viewing A/B Testing experiment data in the Firebase console, you can inspect and analyze experiment data in BigQuery . While A/B Testing does not have a separate BigQuery table, experiment and variant memberships are stored on every Google Analytics event within the Analytics event tables.
The user properties that contain experiment information are of the form userProperty.key like "firebase_exp_%" or userProperty.key = "firebase_exp_01" where 01 is the experiment ID, and userProperty.value.string_value contains the (zero-based) index of the experiment variant.
You can use these experiment user properties to extract experiment data. This gives you the power to slice your experiment results in many different ways and independently verify the results of A/B Testing .
To get started, complete the following as described in this guide:
- Enable BigQuery export for Google Analytics in the Firebase console
- Access A/B Testing data using BigQuery
- Explore example queries
Enable BigQuery export for Google Analytics in the Firebase console
If you're on the Spark plan, you can use the BigQuery sandbox to access BigQuery at no cost, subject to Sandbox limits . See Pricing and the BigQuery sandbox for more information.
First, make sure that you're exporting your Analytics data to BigQuery :
In the Firebase console, go to the
Settings > Integrations tab . In the BigQuery card, click Manage and verify that your project is exporting Analytics data to BigQuery .
If the card says Link , then you need to set up the export (continue to the next step).
If you need to set up export:
Review About Linking Firebase to BigQuery , then click Next .
In the Configure integration section, enable Google Analytics .
Select a region and choose export settings.
Click Link to BigQuery .
Depending on how you chose to export data, it may take up to a day for the tables to become available. For more information about exporting project data to BigQuery , see Export project data to BigQuery .
Access A/B Testing data in BigQuery
Before querying for data for a specific experiment, you'll want to obtain some or all of the following to use in your query:
- Experiment ID: You can obtain this from the URL of the Experiment overview page. For example, if your URL looks like
https://console.firebase.google.com/project/my_firebase_project/config/experiment/results/25, the experiment ID is 25 . - Google Analytics property ID : This is your 9-digit Google Analytics property ID. You can find this within Google Analytics ; it also appears in BigQuery when you expand your project name to show the name of your Google Analytics event table (
project_name.analytics_000000000.events). - Experiment date: To compose a faster and more efficient query, it's good practice to limit your queries to the Google Analytics daily event table partitions that contain your experiment data—tables identified with a
YYYYMMDDsuffix. So, if your experiment ran from February 2, 2024 through May 2, 2024, you'd specify a_TABLE_SUFFIX between '20240202' AND '20240502'. For an example, see Select a specific experiment's values . - Event names: Typically, these correspond with your goal metrics that you configured in the experiment. For example,
in_app_purchaseevents,ad_impression, oruser_retentionevents.
After you gather the information you need to generate your query:
- In the Google Cloud console, go to BigQuery .
- Select your project, then select Create SQL query .
- Add your query. For example queries to run, see Explore example queries .
- Click Run .
Query experiment data using the Firebase console's auto-generated query
If you're using the Blaze plan, the Experiment overview page provides a sample query that returns the experiment name, variants, event names, and the number of events for the experiment you're viewing.
To obtain and run the auto-generated query:
- In the Firebase console, go to DevOps & Engagement > A/B Testing .
- Select the A/B Testing experiment you want to query to open the Experiment overview .
- From the Options menu, beneath BigQuery integration , select Query experiment data . This opens your project in BigQuery within the Google Cloud console console and provides a basic query you can use to query your experiment data.
The following example shows a generated query for an experiment with three variants (including the baseline) named "Winter welcome experiment." It returns the active experiment name, variant name, unique event, and event count for each event. Note that the query builder doesn't specify your project name in the table name, as it opens directly within your project.
/*
This query is auto-generated by Firebase A/B Testing for your
experiment "Winter welcome experiment".
It demonstrates how you can get event counts for all Analytics
events logged by each variant of this experiment's population.
*/
SELECT
'Winter welcome experiment' AS experimentName,
CASE userProperty.value.string_value
WHEN '0' THEN 'Baseline'
WHEN '1' THEN 'Welcome message (1)'
WHEN '2' THEN 'Welcome message (2)'
END AS experimentVariant,
event_name AS eventName,
COUNT(*) AS count
FROM
`analytics_000000000.events_*`,
UNNEST(user_properties) AS userProperty
WHERE
(_TABLE_SUFFIX BETWEEN '20240202' AND '20240502')
AND userProperty.key = 'firebase_exp_25'
GROUP BY
experimentVariant, eventName
For additional query examples, proceed to Explore example queries .
Explore example queries
The following sections provide examples of queries you can use to extract A/B Testing experiment data from Google Analytics event tables.
Extract purchase and experiment standard deviation values from all experiments
You can use experiment results data to independently verify Firebase A/B Testing results. The following BigQuery SQL statement extracts experiment variants, the number of unique users in each variant, and sums total revenue from in_app_purchase and ecommerce_purchase events, and standard deviations for all experiments within the time range specified as the _TABLE_SUFFIX begin and end dates. You can use the data you obtain from this query with a statistical significance generator for one-tailed t-tests to verify that the results Firebase provides match your own analysis.
For more information about how A/B Testing calculates inference, see Interpret test results .
/*
This query returns all experiment variants, number of unique users,
the average USD spent per user, and the standard deviation for all
experiments within the date range specified for _TABLE_SUFFIX.
*/
SELECT
experimentNumber,
experimentVariant,
COUNT(*) AS unique_users,
AVG(usd_value) AS usd_value_per_user,
STDDEV(usd_value) AS std_dev
FROM
(
SELECT
userProperty.key AS experimentNumber,
userProperty.value.string_value AS experimentVariant,
user_pseudo_id,
SUM(
CASE
WHEN event_name IN ('in_app_purchase', 'ecommerce_purchase')
THEN event_value_in_usd
ELSE 0
END) AS usd_value
FROM `PROJECT_NAME.analytics_ANALYTICS_ID.events_*`
CROSS JOIN UNNEST(user_properties) AS userProperty
WHERE
userProperty.key LIKE 'firebase_exp_%'
AND event_name IN ('in_app_purchase', 'ecommerce_purchase')
AND (_TABLE_SUFFIX BETWEEN 'YYYYMMDD' AND 'YYYMMDD')
GROUP BY 1, 2, 3
)
GROUP BY 1, 2
ORDER BY 1, 2;
Select a specific experiment's values
The following example query illustrates how to obtain data for a specific experiment in BigQuery . This sample query returns the experiment name, variant names (including Baseline), event names, and event counts.
SELECT
'EXPERIMENT_NAME' AS experimentName,
CASE userProperty.value.string_value
WHEN '0' THEN 'Baseline'
WHEN '1' THEN 'VARIANT_1_NAME'
WHEN '2' THEN 'VARIANT_2_NAME'
END AS experimentVariant,
event_name AS eventName,
COUNT(*) AS count
FROM
`analytics_ANALYTICS_PROPERTY.events_*`,
UNNEST(user_properties) AS userProperty
WHERE
(_TABLE_SUFFIX BETWEEN 'YYYMMDD' AND 'YYYMMDD')
AND userProperty.key = 'firebase_exp_EXPERIMENT_NUMBER'
GROUP BY
experimentVariant, eventName
When you use Firebase Remote Config to deploy settings for an application with an active user base, you want to make sure you get it right. You can use A/B Testing experiments to best determine the following:
- The best way to implement a feature to optimize the user experience. Too often, app developers don't learn that their users dislike a new feature or an updated user experience until their app's rating in the app store declines. A/B Testing can help measure whether your users like new variants of features, or whether they prefer the app as it exists. Plus, keeping most of your users in a baseline group ensures that most of your user base can continue to use your app without experiencing any changes to its behavior or appearance until the experiment has concluded.
- The best way to optimize the user experience for a business goal. Sometimes you're implementing product changes to maximize a metric like revenue or retention. With A/B Testing , you set your business objective, and Firebase performs the statistical analysis to determine if a variant is outperforming the baseline for your selected objective.
To A/B test feature variants with a baseline, do the following:
- Create your experiment.
- Manage your experiment.
یک آزمایش ایجاد کنید
A Remote Config experiment lets you evaluate multiple variants on one or more Remote Config parameters .
Verify that Google Analytics is enabled in your project so that the experiment has access to Analytics data.
If you didn't enable Google Analytics when creating your project, you can enable it in the
Settings > Integrations tab of the Firebase console. In the Firebase console, go to DevOps & Engagement > A/B Testing
Click Create experiment , and then select Remote Config when prompted for the service you want to experiment with.
In the Variants section, choose a baseline and at least one variant for the experiment. You can add one or more parameters to experiment with. You can repeat this step to add multiple parameters to your experiment.
(optional) To add more than one variant to your experiment, click Add another variant .
Change one or more parameters for specific variants. Any unchanged parameters are the same for users not included in the experiment.
Expand Variant Weights to view or change variant weight for the experiment. By default, each variant is weighted equally. Note that uneven weights may increase data collection time and weights cannot be changed after the experiment begins .
Define the Targeting criteria for your experiment using Remote Config conditions:
Reuse an existing condition: If an existing condition in your Remote Config template already matches your target audience, select it from the list.
Verify condition evaluation order: Make sure your conditions on the Conditions page are organized in the correct priority order. Because Remote Config evaluates conditions sequentially from top to bottom, other higher-priority condition(s) can prevent sufficient users from reaching the condition associated with your experiment.
Create a new condition: If no existing condition satisfies your targeting requirements, or if you prefer to duplicate an existing condition (for example, to avoid locking a condition that is shared by other parameters), create a new condition by first choosing the app that uses your experiment. If the parameter you are testing also uses the existing (or another overlapping) condition, make sure the new experiment condition is placed above the existing condition on the Conditions tab (giving it higher evaluation priority); otherwise, users will match the existing condition first and won't flow into the experiment.
You can then target a specific subset of users by clicking and and selecting one or more options from the following list:
- Version: One or more versions of your app
- Build number: The build number (Apple) or version code (Android) of your app
- Platform: One or more platforms (iOS, Android, or Web) to target
- Operating system: Target web app users based on their operating system and version
- Browser: Target web app users based on their web browser and browser version
- Device category: Target web app users based on whether their device is mobile or non-mobile
- Languages: One or more languages and locales used to select users who might be included in the experiment
- Country/Region: One or more countries or regions for selecting users who should be included in the experiment
- User audience: Analytics audiences used to target users who might be included in the experiment
- User property: One or more Analytics user properties for selecting users who might be included in the experiment
- User in random percentage: Target a randomly selected percentage of users within a defined percentile range
- Imported segment: Target users who belong to custom imported segments uploaded to your project
- Date/Time: Target users based on a specified date and time window
- First open: Target users based on the first time they ever opened your app
- Installation ID: Target specific test devices or client instances using their Firebase Installation IDs (FIDs)
- User exists: Target all users across all apps in the project
- Custom signal: Target users based on custom client-side key-value signals passed at runtime
Set the Exposure: Enter the percentage of your app's user base matching the criteria set under Target users that you want to evenly divide between the baseline and one or more variants in your experiment. This can be any percentage between 0% and 100%. Users are randomly assigned to each experiment, including duplicated experiments.
Optionally, set an activation event to ensure that only the data from users who have first triggered some Analytics event are counted in your experiment. Note that all users matching your targeting parameters will receive Remote Config experimental values, but only those who trigger an activation event will be included in your experiment results.
To ensure a valid experiment, make sure that the event you choose occurs after your app activates fetched configuration values. In addition, the following events cannot be used because they always occur before fetched values are activated:
-
app_install -
app_remove -
app_update
The Analytics event you select as the activation event must not also be used as the primary metric (or as an additional metric) in the same experiment. Doing so will trigger a validation error in the Firebase console and prevent your experiment from launching.
-
For the experiment's Goals , select the primary metric to track, and add any additional metrics you want to track from the list. These include built-in objectives (purchases, revenue, retention, crash-free users, etc.), Analytics conversion events, and other Analytics events. When finished, click Next .
Click Save to save your experiment. You must publish the template to start running your experiment.
You are allowed up to 300 experiments per project (including rollouts), which could consist of up to 24 running experiments and rollouts, with the rest as completed experiments.
Manage your experiment
When you create an experiment with Remote Config , you can start your experiment, monitor your experiment while it is running, and increase the number of users included in your running experiment.
When your experiment is done, you can take note of the settings used by the winning variant, and then roll out those settings to all users. Or, you can run another experiment.
Edit an experiment
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
- Click Running , click an experiment that you want to edit.
- Click the context menu ( ) and click Edit running experiment .
- To validate that your app has users who would be included in your experiment, expand the details and check for a number greater than 0% in the Targeting and distribution section (for example, 1% of users matching the criteria ).
Monitor an experiment
Once an experiment has been running for a while, you can check in on its progress and see what your results look like for the users who have participated in your experiment so far.
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
Click Running , and then click, or search for, the title of your experiment. On this page, you can view various observed and modeled statistics about your running experiment, including the following:
- % difference from baseline : A measure of the improvement of a metric for a given variant as compared to the baseline. Calculated by comparing the value range for the variant to the value range for the baseline.
- Probability to beat baseline : The estimated probability that a given variant beats the baseline for the selected metric.
- observed_metric per user : Based on experiment results, this is the predicted range that the metric value will fall into over time.
- Total observed_metric : The observed cumulative value for the baseline or variant. The value is used to measure how well each experiment variant performs, and is used to calculate Improvement , Value range , Probability to beat baseline , and Probability to be the best variant . Depending on the metric being measured, this column may be labeled "Duration per user," "Revenue per user," "Retention rate," or "Conversion rate."
After your experiment has run for a while (14 days for Remote Config ), data on this page indicates which variant, if any, is the "leader." Some measurements are accompanied by a bar chart that presents the data in a visual format.
Roll out an experiment to all users
After an experiment has run long enough that you have a "leader," or winning variant, for your goal metric, you can release the experiment to 100% of users. This lets you select a variant to publish to all users moving forward. Even if your experiment has not created a clear winner, you can still choose to release a variant to all of your users.
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
- Click Completed or Running , click an experiment that you want to release to all users, click the context menu ( ) Roll out variant .
- Roll out your experiment to all users by doing the following:
- For a Remote Config experiment, select a variant to determine which Remote Config parameter values to update. The targeting criteria defined when creating the experiment is added as a new condition in your template, to ensure the rollout only affects users targeted by the experiment. After clicking Review in Remote Config to review the changes, click Publish changes to complete the rollout.
Expand an experiment
If you find that an experiment isn't bringing in enough users for A/B Testing to declare a leader, you can increase distribution of your experiment to reach a larger percentage of the app's user base.
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
- Select the running experiment that you want to edit.
- In the Experiment overview , click the context menu ( ), and then click Edit running experiment .
- The Targeting dialog displays an option to increase the percentage of users who are in the running experiment. Select a number greater than the current percentage and click Publish . The experiment will be pushed out to the percentage of users you have specified.
Duplicate an experiment
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
- Select the running or completed experiment that you want to stop.
- Click Completed or Running , hold the pointer over your experiment, click the context menu ( ), and then click Duplicate experiment or Stop experiment .
Stop an experiment
- In the DevOps & Engagement section of the Firebase console navigation menu, click Remote Config .
- Click the A/B Tests tab.
- Select the running or completed experiment that you want to stop.
- Click Completed or Running , hold the pointer over your experiment, click the context menu ( ), and then click Stop experiment .
Web client identification and experiment persistence
When a user launches a web application using Firebase A/B Testing in a browser for the first time, a unique Firebase installation ID ( FID) is generated. This FID is persistently stored in the browser's IndexedDB to identify the app instance across sessions.
Firebase A/B Testing uses the FID to assign users to experiment variants, and Google Analytics uses it for event aggregation to measure and analyze user behavior within each variant.
Because the FID is stored in IndexedDB, Firebase A/B Testing treats a user as a new user if they access your app from a different browser or in an incognito window, or if they clear their browser's IndexedDB. This means that a user might be included in different experiment variants when using different browsers or browsing sessions.
User targeting
You can target the users to include in your experiment using the following user-targeting criteria.
The following rule types are supported in the Firebase console. Equivalent features are available in the Remote Config REST API, as detailed in the conditional expression reference .
| Rule type | اپراتور(ها) | ارزش(ها) | توجه داشته باشید |
| برنامه | == | Select from a list of App IDs for apps associated with your Firebase project. | When you add an app to Firebase, you enter a bundle ID or Android package name that defines an attribute that's exposed as App ID in Remote Config rules. Use this attribute as follows:
|
| نسخه برنامه | For string values: exactly matches, contains, does not contain, contains regular expression For numeric values: <, <=, =, !=, >, >= | Specify the version(s) of your app to target. Before using this rule, you must use an App ID rule to select an Android/Apple app associated with your Firebase project. | For Apple platforms: Use the app's CFBundleShortVersionString . Note: Make sure your Apple app is using Firebase Apple platforms SDK version 6.24.0 or higher, as CFBundleShortVersionString is not being sent in earlier versions (see release notes ). For Android: Use the app's versionName . Note: Numeric comparison operators ( String comparisons for this rule are case-sensitive. When using the exactly matches , contains , does not contain , or contains regular expression operator, you can select multiple values. When using the contains regular expression operator, you can create regular expressions in RE2 format. Your regular expression can match all or part of the target version string. You can also use the ^ and $ anchors to match the beginning, end, or entirety of a target string. |
| شماره ساخت | For string values: exactly matches, contains, does not contain, عبارت منظم For numeric values: =, ≠, >, ≥, <, ≤ | Specify the build(s) of your app to target. Before using this rule, you must use an App ID rule to select an Apple or Android app associated with your Firebase project. | This operator is available for Apple and Android apps only. It corresponds to the app's CFBundleVersion for Apple and versionCode for Android. String comparisons for this rule are case-sensitive. When using the exactly matches , contains , does not contain , or contains regular expression operator, you can select multiple values. When using the contains regular expression operator, you can create regular expressions in RE2 format. Your regular expression can match all or part of the target version string. You can also use the ^ and $ anchors to match the beginning, end, or entirety of a target string. |
| پلتفرم | == | آیاواس اندروید وب | |
| سیستم عامل | == | Specify the operating system(s) to target. Before using this rule, you must use an App ID rule to select a Web app associated with your Firebase project. | This rule evaluates to true for a given Web app instance if the operating system and its version matches a target value in the specified list. |
| مرورگر | == | Specify the browser(s) to target. Before using this rule, you must use an App ID rule to select a Web app associated with your Firebase project. | This rule evaluates to true for a given Web app instance if the browser and its version matches a target value in the specified list. |
| Device category | is, is not | موبایل | This rule evaluates whether the device accessing your web app is mobile or non-mobile (desktop or console). This rule type is only available for web apps. |
| زبانها | در است | Select one or more languages. | This rule evaluates to true for a given app instance if that app instance is installed on a device that uses one of the languages listed. |
| کشور/منطقه | در است | Select one or more regions or countries. | This rule evaluates to true for a given app instance if the instance is in any of the regions or countries listed. The device country code is determined using the device's IP address in the request or the country code determined by Firebase Analytics (if Analytics data is shared with Firebase). |
| User audience(s) | شامل حداقل یکی است | Select one or more from a list of Google Analytics audiences that you have set up for your project. | This rule requires an App ID rule to select an app associated with your Firebase project. Note: Because many Analytics audiences are defined by events or user properties, which can be based on the actions of app users, it may take some time for a User in audience rule to take effect for a given app instance. This means that even if a user technically qualifies for an audience, if Analytics has not yet added the user to the audience when |
| User property | For string values: contains, does not contain, exactly matches, contains regular expression For numeric values: =, ≠, >, ≥, <, ≤ Note: On the client, you can set only string values for user properties. For conditions that use numeric operators, Remote Config converts the value of the corresponding user property into an integer/float. | Select from a list of available Google Analytics user properties. | To learn how you can use user properties to customize your app for very specific segments of your user base, see Remote Config and user properties . To learn more about user properties, see the following guides: When using the exactly matches , contains , does not contain or contains regular expression operator, you can select multiple values. When using the contains regular expression operator, you can create regular expressions in RE2 format. Your regular expression can match all or part of the target version string. You can also use the ^ and $ anchors to match the beginning, end, or entirety of a target string. Note: Automatically collected user properties are not available when creating Remote Config conditions. |
| User in random percentage | Slider (in the Firebase console. The REST API uses the <= , > , and between operators). | ۰-۱۰۰ | Use this field to apply a change to a random sample of app instances (with sample sizes as small as .0001%), using the slider widget to segment randomly-shuffled users (app instances) into groups. Each app instance is persistently mapped to a random whole or fractional number, according to a seed defined in that project. A rule will use the default key (shown as Edit seed in the Firebase console) unless you modify the seed value. You can return a rule to using the default key by clearing the Seed field. To consistently address the same app instances within given percentage ranges, use the same seed value across conditions. Or, select a new randomly-assigned group of app instances for a given percentage range by specifying a new seed. For example, to create two related conditions that each apply to a non-overlapping 5% of an app's users, you could configure one condition to match a percentage between 0% and 5% and configure another condition to match a range between 5% and 10%. To allow some users to randomly appear in both groups, use different seed values for the rules within each condition. |
| Imported segment | در است | Select one or more imported segment. | This rule requires setting up custom imported segments . |
| تاریخ/زمان | قبل، بعد | A specified date and time, either in the device timezone or a specified timezone such as "(GMT+11) Sydney time." | Compares the current time to the device fetch time. |
| First open | قبل، بعد | Target users based on the first time they open your app:
| User targeting by first open is available after you select an Android, iOS, or Web app. Requires the following SDKs:
Analytics must also have been enabled on the client during the first open event. |
| شناسه نصب | در است | Specify one or more Installation IDs (up to 50) to target. | This rule evaluates to true for a given installation if that installation's ID is in the comma-separated list of values.To learn how you can get installation IDs, see Retrieve client identifiers . |
| User exists | (no operator) | Targets all users of all apps within the current project. | Use this condition rule to match all users within the project, regardless of app or platform. |
| سیگنال سفارشی | For string values: contains, does not contain, exactly matches, contains regular expression For numeric values: =, ≠, >, ≥, <, ≤ For version values: =, ≠, >, ≥, <, ≤ | String comparisons for this rule are case-sensitive. When using the exactly matches, contains, does not contain, or contains regular expression operator, you can select multiple values. When using the contains regular expression operator, you can create regular expressions in RE2 format. Your regular expression can match all or part of the target version string. You can also use the ^ and $ anchors to match the beginning, end, or entirety of a target string. The following data types are supported for client environments:
Numeral that represents the version number(s) to match (for example, 2.1.0). | For more information on custom signal conditions and conditional expressions to use, see Custom signal conditions and Elements used to create conditions . |
A/B Testing metrics
When you create your experiment, you choose a primary, or goal metric, that is used to determine the winning variant. You should also track other metrics to help you better understand each experiment variant's performance and track important trends that may differ for each variant, like user retention, app stability and in-app purchase revenue. You can track up to five non-goal metrics in your experiment.
For example, say you're using Remote Config to launch two different game flows in your app and want to optimize for in-app purchases and ad revenue, but you also want to track the stability and user retention of each variant. In this case, you might consider choosing Estimated total revenue as your goal metric because it includes in-app purchase revenue and ad revenue, and then, for Other metrics to track , you might add the following:
- To track your daily and weekly user retention, add Retention (2-3 days) and Retention (4-7 days) .
- To compare stability between the two game flows, add Crash-free users .
- To see more detailed views of each revenue type, add Purchase revenue and Estimated ad revenue .
The following tables provide details on how goal metrics and other metrics are calculated.
Goal metrics
| متریک | توضیحات |
|---|---|
| Crash-free users | The percentage of users who have not encountered errors in your app that were detected by the Firebase Crashlytics SDK during the experiment. Note: Firebase Crashlytics is not supported for web applications. |
| Estimated ad revenue | Estimated ad earnings. |
| Estimated total revenue | Combined value for purchase and estimated ad revenues. |
| Purchase revenue | Combined value for all purchase and in_app_purchase events. |
| Retention (1 day) | The number of users who return to your app on a daily basis. |
| Retention (2-3 days) | The number of users who return to your app within 2-3 days. |
| Retention (4-7 days) | The number of users who return to your app within 4-7 days. |
| Retention (8-14 days) | The number of users who return to your app within 8-14 days. |
| Retention (15+ days) | The number of users who return to your app 15 or more days after they last used it. |
| first_open | An Analytics event that triggers when a user first opens an app after installing or reinstalling it. Used as part of a conversion funnel. |
سایر معیارها
| متریک | توضیحات |
|---|---|
| notification_dismiss | An Analytics event that triggers when a notification sent by the Notifications composer is dismissed (Android only). |
| notification_receive | An Analytics event that triggers when a notification sent by the Notifications composer is received while the app is in the background (Android only). |
| os_update | An Analytics event that tracks when the device operating system is updated to a new version.To learn more, see Automatically collected events . This metric is not supported for web applications. |
| screen_view | An Analytics event that tracks screens viewed within your app. To learn more, see Track Screenviews . |
| session_start | An Analytics event that counts user sessions in your app. To learn more, see Automatically collected events . |
BigQuery data export
In addition to viewing A/B Testing experiment data in the Firebase console, you can inspect and analyze experiment data in BigQuery . While A/B Testing does not have a separate BigQuery table, experiment and variant memberships are stored on every Google Analytics event within the Analytics event tables.
The user properties that contain experiment information are of the form userProperty.key like "firebase_exp_%" or userProperty.key = "firebase_exp_01" where 01 is the experiment ID, and userProperty.value.string_value contains the (zero-based) index of the experiment variant.
You can use these experiment user properties to extract experiment data. This gives you the power to slice your experiment results in many different ways and independently verify the results of A/B Testing .
To get started, complete the following as described in this guide:
- Enable BigQuery export for Google Analytics in the Firebase console
- Access A/B Testing data using BigQuery
- Explore example queries
Enable BigQuery export for Google Analytics in the Firebase console
If you're on the Spark plan, you can use the BigQuery sandbox to access BigQuery at no cost, subject to Sandbox limits . See Pricing and the BigQuery sandbox for more information.
First, make sure that you're exporting your Analytics data to BigQuery :
In the Firebase console, go to the
Settings > Integrations tab . In the BigQuery card, click Manage and verify that your project is exporting Analytics data to BigQuery .
If the card says Link , then you need to set up the export (continue to the next step).
If you need to set up export:
Review About Linking Firebase to BigQuery , then click Next .
In the Configure integration section, enable Google Analytics .
Select a region and choose export settings.
Click Link to BigQuery .
Depending on how you chose to export data, it may take up to a day for the tables to become available. For more information about exporting project data to BigQuery , see Export project data to BigQuery .
Access A/B Testing data in BigQuery
Before querying for data for a specific experiment, you'll want to obtain some or all of the following to use in your query:
- Experiment ID: You can obtain this from the URL of the Experiment overview page. For example, if your URL looks like
https://console.firebase.google.com/project/my_firebase_project/config/experiment/results/25, the experiment ID is 25 . - Google Analytics property ID : This is your 9-digit Google Analytics property ID. You can find this within Google Analytics ; it also appears in BigQuery when you expand your project name to show the name of your Google Analytics event table (
project_name.analytics_000000000.events). - Experiment date: To compose a faster and more efficient query, it's good practice to limit your queries to the Google Analytics daily event table partitions that contain your experiment data—tables identified with a
YYYYMMDDsuffix. So, if your experiment ran from February 2, 2024 through May 2, 2024, you'd specify a_TABLE_SUFFIX between '20240202' AND '20240502'. For an example, see Select a specific experiment's values . - Event names: Typically, these correspond with your goal metrics that you configured in the experiment. For example,
in_app_purchaseevents,ad_impression, oruser_retentionevents.
After you gather the information you need to generate your query:
- In the Google Cloud console, go to BigQuery .
- Select your project, then select Create SQL query .
- Add your query. For example queries to run, see Explore example queries .
- Click Run .
Query experiment data using the Firebase console's auto-generated query
If you're using the Blaze plan, the Experiment overview page provides a sample query that returns the experiment name, variants, event names, and the number of events for the experiment you're viewing.
To obtain and run the auto-generated query:
- In the Firebase console, go to DevOps & Engagement > A/B Testing .
- Select the A/B Testing experiment you want to query to open the Experiment overview .
- From the Options menu, beneath BigQuery integration , select Query experiment data . This opens your project in BigQuery within the Google Cloud console console and provides a basic query you can use to query your experiment data.
The following example shows a generated query for an experiment with three variants (including the baseline) named "Winter welcome experiment." It returns the active experiment name, variant name, unique event, and event count for each event. Note that the query builder doesn't specify your project name in the table name, as it opens directly within your project.
/*
This query is auto-generated by Firebase A/B Testing for your
experiment "Winter welcome experiment".
It demonstrates how you can get event counts for all Analytics
events logged by each variant of this experiment's population.
*/
SELECT
'Winter welcome experiment' AS experimentName,
CASE userProperty.value.string_value
WHEN '0' THEN 'Baseline'
WHEN '1' THEN 'Welcome message (1)'
WHEN '2' THEN 'Welcome message (2)'
END AS experimentVariant,
event_name AS eventName,
COUNT(*) AS count
FROM
`analytics_000000000.events_*`,
UNNEST(user_properties) AS userProperty
WHERE
(_TABLE_SUFFIX BETWEEN '20240202' AND '20240502')
AND userProperty.key = 'firebase_exp_25'
GROUP BY
experimentVariant, eventName
For additional query examples, proceed to Explore example queries .
Explore example queries
The following sections provide examples of queries you can use to extract A/B Testing experiment data from Google Analytics event tables.
Extract purchase and experiment standard deviation values from all experiments
You can use experiment results data to independently verify Firebase A/B Testing results. The following BigQuery SQL statement extracts experiment variants, the number of unique users in each variant, and sums total revenue from in_app_purchase and ecommerce_purchase events, and standard deviations for all experiments within the time range specified as the _TABLE_SUFFIX begin and end dates. You can use the data you obtain from this query with a statistical significance generator for one-tailed t-tests to verify that the results Firebase provides match your own analysis.
For more information about how A/B Testing calculates inference, see Interpret test results .
/*
This query returns all experiment variants, number of unique users,
the average USD spent per user, and the standard deviation for all
experiments within the date range specified for _TABLE_SUFFIX.
*/
SELECT
experimentNumber,
experimentVariant,
COUNT(*) AS unique_users,
AVG(usd_value) AS usd_value_per_user,
STDDEV(usd_value) AS std_dev
FROM
(
SELECT
userProperty.key AS experimentNumber,
userProperty.value.string_value AS experimentVariant,
user_pseudo_id,
SUM(
CASE
WHEN event_name IN ('in_app_purchase', 'ecommerce_purchase')
THEN event_value_in_usd
ELSE 0
END) AS usd_value
FROM `PROJECT_NAME.analytics_ANALYTICS_ID.events_*`
CROSS JOIN UNNEST(user_properties) AS userProperty
WHERE
userProperty.key LIKE 'firebase_exp_%'
AND event_name IN ('in_app_purchase', 'ecommerce_purchase')
AND (_TABLE_SUFFIX BETWEEN 'YYYYMMDD' AND 'YYYMMDD')
GROUP BY 1, 2, 3
)
GROUP BY 1, 2
ORDER BY 1, 2;
Select a specific experiment's values
The following example query illustrates how to obtain data for a specific experiment in BigQuery . This sample query returns the experiment name, variant names (including Baseline), event names, and event counts.
SELECT
'EXPERIMENT_NAME' AS experimentName,
CASE userProperty.value.string_value
WHEN '0' THEN 'Baseline'
WHEN '1' THEN 'VARIANT_1_NAME'
WHEN '2' THEN 'VARIANT_2_NAME'
END AS experimentVariant,
event_name AS eventName,
COUNT(*) AS count
FROM
`analytics_ANALYTICS_PROPERTY.events_*`,
UNNEST(user_properties) AS userProperty
WHERE
(_TABLE_SUFFIX BETWEEN 'YYYMMDD' AND 'YYYMMDD')
AND userProperty.key = 'firebase_exp_EXPERIMENT_NUMBER'
GROUP BY
experimentVariant, eventName