1% Xen Release Management 2% Wei Liu <<wei.liu2@citrix.com>> 3% Revision 1 4 5# Motivation 6 7Over the years we have had different people signing up as the Release Manager 8of Xen. It would be rather wasteful if every new Release Manager has to go over 9everything and tripped over by the same mistakes again and again. 10 11This file intends to document the process of managing a Xen release. It is 12mainly written for Release Manager, but other roles (contributors, 13maintainers and committers) are also encouraged to read this document, so 14that they can have an idea what to expect from the Release Manager. 15 16# Xen release cycle 17 18The Xen hypervisor project now releases every 8 months. The actual release date 19depends on a lot of factors. 20 21We can roughly divide one release into two periods. The development period 22and the freeze period. The former is 6 months long and the latter is about 2 23months long. 24 25During development period, contributors submit patches to be reviewed and 26committed into xen.git. All feature patches must be committed before a date, 27which is normally called the "cut-off date", after which the freeze period 28starts. There will be a date before which all patches that wish to be merged 29for the release should be posted -- it is normally called the "last posting 30date" and it is normally two weeks before the "cut-off date". 31 32During freeze period, the tree is closed for new features. Only bug fixes are 33accepted. This period can be shorter or longer than 2 months. If it ends up 34longer than 2 months, it eats into the next development period. 35 36# The different roles in a Xen release 37 38## Release Manager 39 40A trusted developer in the community that owns the release process. The major 41goal of the Release Manager is to make sure a Xen release has high quality 42and doesn't slip too much. 43 44The Release Manager will not see much workload during development period, but 45expects to see increasing workload during the freeze period until the final 46release. He or she is expected to keep track of issues, arrange RCs, 47negotiate with relevant stakeholders, balance the need from various parties 48and make difficult decisions when necessary. 49 50The Release Manager essentially owns xen-unstable branch during the freeze 51period. The Committers will act on the wishes of the Release Manager during 52that time. 53 54## Maintainers 55 56A group of trusted developers who are responsible for certain components in 57xen.git. They are expected to respond to patches / questions with regard to 58their components in a timely manner, especially during the freeze period. 59 60## Committers 61 62A group of trusted maintainers who can commit to xen.git. During the 63development window they normally push things as they see fit. During the 64freeze period they transfer xen-unstable branch ownership and act on the 65wishes of the Release Manager. That normally means they need to have an 66Release Ack in order to push a patch. 67 68## Contributors 69 70Contributors are also expected to respond quickly to any issues regarding the 71code they submitted during development period. Failing that, the Release 72Manager might decide to revert the changes, declare feature unsupported or 73take any action he / she deems appropriate. 74 75## The Security Team 76 77The Security Team operates independently. The visibility might be rather 78limited due to the sensitive nature of security work. The best action the 79Release Manager can take is to set aside some time for potential security 80issues to be fixed. 81 82## The Release Technician 83 84The Release Technician is the person who tags various trees, prepares tarball 85etc. He or she acts on the wishes of the Release Manager. Please make sure 86the communication is as clear as it can be. 87 88## The Community Manager 89 90The Community Manager owns xenproject.org infrastructure. He or she is 91responsible for updating various web archives, updating wiki pages and 92coordinating with the PR Personnel. 93 94## The PR Personnel 95 96They are responsible for coordinating with external reporters to publish Xen 97release announcement. The Release Manager should be absolutely sure the 98release is going out on a particular date before giving them the signal to 99proceed, because there is a point of no return once they schedule a date with 100external reporters. 101 102# What happens during a release 103 104## Development period 105 106Send out monthly update email. The email contains the timeline of the 107release, the major work items and any other information the Release Manager 108sees fit. Reminders should also be sent one week before important dates (see 109above, "last posting date" and "cut-off date"). Please consider adding 110relevant events to your calendar. 111 112Occasionally check the status of the xen-unstable branch, make sure it gets 113timely pushes to master. 114 115## Freeze period 116 117Before or at very early stage of the freeze period, agree with the Community 118Manager a schedule for RC test days. 119 120Once the freeze starts, the ownership of xen-unstable branch automatically 121transfers to the Release Manager. The Release Manager can say "not releasing 122now" because of too many bugs, "until someone fixes these", or "no more 123patches until X, Y, and Z happen". 124 125Here is a list of things to do for making RCs: 126 1271. Check the status of the tree. Ask the Release Technician to make an RC if 128the tree is good. 129 1302. Send an email to xen-devel, xen-users and xen-announce to announce the RC. 131 1323. Branch and / or reopen the tree for further feature submission if 133appropriate. 134 1354. Collect and track any issues reported, determine their severity, prod 136relevant developers and maintainers to fix the issues. 137 1385. When patches to fix issues are posted, determine if the patches are good to 139be included. 140 1416. Go back to 1. 142 143It is normally OK in the early RCs that you hand back xen-unstable branch to 144committers so that they can commit bug fixes at will. As we approach late 145RCs, the standard for accepting a patch will get higher and higher. Please 146communicate clearly when committers can commit at will and when formal 147Release Ack is needed. 148 149At the same time, work with the Community Manager, PR Personnel and 150Contributors to gather a list of features for the release. Discuss the 151support status of new features with stakeholders. Help prepare the press 152release, write a blog post for the release. 153 154Make sure the key people for doing the release (especially Community Manager, 155Release Manager and Release Technician) will be either available around the 156planned release date or have named a substitute being capable to perform the 157required tasks. 158 1591. Collate a list of major changes: this should be done in collaboration 160between Release Manager, PR Personnel and key contributors. This should *not* 161be done on a public mailing list, to minimize the risk of release related 162media stories being published before the release date. 163 1642. PR Personnel will identify feature highlights, a theme for the press 165release, companies providing supporting quotes for the press release and 166media outlets we would want to reach out to and will manage the creation of 167the press release in private. 168 1693. The Community Manager will also draft blog post with the help of PR 170Personnel and Release Manager, which will be published under the name of the 171Release Manager. 172 1734. The Community Manager will create release related documentation such as 174Acknowledgements, Feature List, Man Pages and Release Notes on the wiki 175accessible via a release category. This can be done in public. 176 1775. PR Personnel will get stake-holder and Advisory Board approval for the 178press release (1-2 weeks before the release). 179 180 181When you think all pending issues are fixed and Xen is ready to be released 182from the last RC: 183 1841. Send out commit moratorium emails to committers@. 185 1862. Check all the trees (mini-os, qemu-trad, qemu-xen, seabios, ovmf etc). 187They have the correct commits and all security patches applied. There will be 188tools provided. 189 1903. Negotiate release date options with PR personnel. Typically we need 3-4 191days to line up press briefings with reporters under embargo. PR personnel 192will also need to consider industry events to ensure that PR is effective. PR 193releases typically done mid-week (Tuesday - Thursday). 194 1954. Select the release date. 196 1975. Specify the dates regarding support and security support in SUPPORT.md. 198 1996. Check with relevant stake-holders (typically community manager) whether 200wiki documentation and PR is in good shape (for an example see 201https://wiki.xenproject.org/wiki/Category:Xen_4.9 202<https://wiki.xenproject.org/wiki/Category:Xen_4.9>) 203 2047. Obtain a formal go-ahead from 205 206 * the Community Manager 207 * the Release Technician 208 209 Ask them to dry-run their checklist and confirm everything is OK. If not, 210 arrange another RC and restart this checklist. 211 2128. Do not commit to a release date until 213 214 * The exact xen.git commit id to be released is known. 215 * That commit id has been satisfactorily tested. 216 2179. Give PR Personnel final go-ahead, and instruct Release Technician to make 218release deliverables (tags and tarballs - will usually be in place the day 219before the release). At this point, PR collateral will be sent to reporters 220(typically 2-3 working days before the release date) and we cannot undo 221publications without questions being asked and risk of negative PR. It is 222acceptable to make a xen-devel@ announcement *before* the PR release date 223(blog, xen-announce@, press release). 224 22510. Make the announcement on various mailing list, publish the blog post. 226 227Allow for contingencies. It is not uncommon that some last minute (security or 228not) bugs are discovered. To provide a fix takes time, the test of the fix 229will also take time. Allow for at least 1 week from getting a fix to getting 230a push. For security bugs, coordinate with the Security Team to adjust the 231dates according to our security policy. 232 233## Hand over of Release Manager responsibility 234 235If there is a new Release Manager for the next release, make sure the 236following things happen for the new Release Manager. 237 2381. A JIRA (xenproject.atlassian.net) is created and proper permissions granted. 2392. Access to community test infrastructure is granted. 240 In the common case the public pages at logs.test-lab.xenproject.org will 241 suffice. 2423. Access to mailing list moderation panel is granted. 2434. An account for blog.xenproject.org is created. 244 The account can be created by the new Release Manager, it might be necessary 245 to adjust the access rights. 2465. An account for wiki.xenproject.org is created. 247 The account can be created by the new Release Manager, it might be necessary 248 to adjust the access rights. 249 250# Email templates and scripts 251 252Note: if you want specific actions from committers, please make sure you CC 253committers@. 254 255## RC emails 256 257``` 258Subject: Xen X.Y rcZ 259 260Hi all, 261 262Xen X.Y rcZ is tagged. You can check that out from xen.git: 263 264git://xenbits.xen.org/xen.git X.Y.0-rcZ 265 266For your convenience there is also a tarball at: 267https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz 268 269And the signature is at: 270https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz.sig 271 272Please send bug reports and test reports to xen-devel@lists.xenproject.org. 273When sending bug reports, please CC relevant maintainers and me 274(abc@xyz.com). 275 276As a reminder, there will be another Xen Test Day. 277 278See instructions on: URL_TO_TEST_INSTRUCTIONS 279``` 280 281## Forego control of the tree 282 283``` 284Subject: No Release Ack needed before RcX 285 286Committers, 287 288The tree is in good state. No release ack is needed before RcX. Please commit 289bug fixes at will. 290 291$RM 292``` 293 294## Commit moratorium 295 296``` 297Subject: Commit moratorium for $REASON 298 299Committers, 300 301Please don't push any new patch to staging because $REASON. 302 303Another email will be sent once the moratorium is lifted. 304 305$RM 306``` 307 308## Lift commit moratorium 309 310``` 311Subject: Commit moratorium is lifted for $REASON 312 313Committers, 314 315The commit moratorium is lifted, please commit patches that are already 316Release-acked. 317 318$RM 319``` 320 321## Reminder of last posting date 322 323``` 324Subject: Last posting date for Xen X.Y is $DATE 325 326Hi all, 327 328The last posting date for Xen X.Y is $DATE. If you want your features to be 329included for the release, please make sure they are posted for the first 330time before $DATE. 331 332$RM 333``` 334 335## Reminder of cut-off date 336 337``` 338Subject: Cut-off date for Xen X.Y is $DATE 339 340Hi all, 341 342The cut-off date for Xen X.Y is $DATE. If you want your features to be 343included for the release, please make sure they are committed by $DATE. 344 345$RM 346``` 347 348## Release announcement 349 350``` 351 Subject: [ANNOUNCEMENT] Xen X.Y is released 352 353 Dear community members, 354 355 I'm pleased to announce that Xen X.Y.0 is released. 356 357 Please find the tarball and its signature at: 358 359 https://xenproject.org/downloads/xen-archives/xen-project-xy-series/xen-project-xy0.html 360 361 You can also check out the tag in xen.git: 362 363 https://xenbits.xen.org/git-http/xen.git RELEASE-X.Y.0 364 365 Git checkout and build instructions can be found at: 366 367 https://wiki.xenproject.org/wiki/Xen_Project_X.Y_Release_Notes#Build_Requirements 368 369 Release notes can be found at: 370 371 https://wiki.xenproject.org/wiki/Xen_Project_X.Y_Release_Notes 372 373 A summary for X.Y release documents can be found at: 374 375 https://wiki.xenproject.org/wiki/Category:Xen_X.Y 376 377 Technical blog post for X.Y can be found at: 378 379 URL_TO_BLOG 380 381 Thanks everyone who contributed to this release. This release would 382 not have happened without all the awesome contributions from around 383 the globe. 384 385 Regards, 386 387 $RM (on behalf of the Xen Project Hypervisor team) 388``` 389 390 391## Script to generate months update emails 392 393``` 394#!/bin/bash 395# Use ssmtp for simplicity 396# ./status-release.sh | formail -f -s /usr/sbin/ssmtp -bm -t 397 398FILE=`mktemp` 399cat << EOF > $FILE 400 401== Hypervisor == 402 403S: Per-cpu tasklet 404O: Konrad Rzeszutek Wilk 405E: konrad.wilk@oracle.com 406J: XEN-28 407 408=== x86 === 409 410=== ARM === 411 412== Completed == 413 414S: 415EOF 416 417 418AWK_FILE=`mktemp` 419cat << EOF > $AWK_FILE 420BEGIN { s2_count = 1;score = ""; emails=1; first_time = 1; subject=""} 421/== / { 422 if ( subject != "" ) { 423 if (score != "") 424 print "* ", subject, "("score")" 425 else if (version != "") 426 print "* ", subject, "("version")"; 427 else 428 print "* ", subject; 429 for (i = 1; i <= s2_count; i++) { 430 if (i in s2) 431 print " ",s2[i]; 432 } 433 if (bug != "") 434 print " Link: https://bugs.xenproject.org/xen/bug/"bug 435 if (jira != "") 436 print " - "jira 437 for (i = 1; i <= count; i++) { 438 if (i in o) 439 print " -", o[i] 440 } 441 if (emails) 442 print "" 443 first_time = 1; 444 subject="" 445 email="" 446 score="" 447 bug="" 448 jira="" 449 version="" 450 count = 1; 451 s2_count = 1; 452 delete s; 453 delete s2; 454 delete o; 455 delete e; 456 } 457 print \$0,"\n" 458 } 459/;/ { }; 460/S:/ { 461 if ( !first_time ) { 462 if (score != "") 463 print "* ", subject, "("score")" 464 else if (version != "") 465 print "* ", subject, "("version")"; 466 else 467 print "* ", subject 468 for (i = 1; i <= s2_count; i++) { 469 if (i in s2) 470 print " ",s2[i]; 471 } 472 if (bug != "") 473 print " Link: https://bug.xenproject.org/xen/bug/"bug 474 if (jira != "") 475 print " - "jira 476 for (i = 1; i <= count; i++) { 477 if (i in o) 478 print " -", o[i] 479 } 480 if (emails) 481 print "" 482 } 483 first_time = 0; 484 sub(\$1, ""); 485 sub(/^[ \t]+/, ""); 486 subject=\$0; 487 email="" 488 bug="" 489 jira="" 490 count = 1; 491 s2_count = 1; 492 delete s; 493 delete s2; 494 delete o; 495 delete e; 496 score=""; 497 version=""; 498 } 499/O:/ { sub(\$1, ""); o[count++]=\$0; }; 500/S2:/ { sub(\$1, ""); s2[s2_count++]=\$0;}; 501/E:/ { sub(\$1, ""); sub(/^[ \t]+/, ""); email=\$0; e[emails++]=\$0;}; 502/P:/ { sub(\$1, ""); sub(/^[ \t]+/, ""); score=\$0; }; 503/B:/ { sub(\$1, ""); sub(/^[ \t]+/, ""); bug=\$0; }; 504/J:/ { sub(\$1, ""); sub(/^[ \t]+/, ""); jira=\$0; }; 505/V:/ { sub(\$1, ""); sub(/^[ \t]+/, ""); version=\$0; }; 506END { 507 } 508// { } 509EOF 510AWK_FILE_EMAIL=`mktemp` 511cat << EOF > $AWK_FILE_EMAIL 512BEGIN { emails=1;} 513/E:/ { 514 sub(\$1, ""); sub(/^[ \t]+/, ""); 515 email=\$0; 516 for ( i = 1; i <= emails; i++ ) { 517 if (i in e) { 518 if (e[i] == email) { 519 email=""; 520 break; 521 } 522 } 523 } 524 if (email != "") 525 e[emails++]=email; 526} 527END { 528 printf "Bcc: " 529 for ( i = 1; i <= emails; i++ ) 530 if (i in e) { 531 if (i == emails - 1) 532 printf "<%s>", e[i]; 533 else 534 printf "<%s>,", e[i]; 535 } 536 print "" 537 } 538// { } 539EOF 540 541echo "From: $RELEASE_MANAGER_NAME <$RELEASE_MANAGER_MAIL>" 542echo "To: xen-devel@lists.xenproject.org" 543echo "Cc: $RELEASE_MANAGER_MAIL" 544cat $FILE | awk -f $AWK_FILE_EMAIL 545rm $AWK_FILE_EMAIL 546 547echo "Subject: Xen $RELEASE_VERSION Development Update" 548PRE=`mktemp` 549cat << EOF > $PRE 550 551This email only tracks big items for xen.git tree. Please reply for items you 552would like to see in $RELEASE_VERSION so that people have an idea what is going on and 553prioritise accordingly. 554 555You're welcome to provide description and use cases of the feature you're 556working on. 557 558= Timeline = 559 560We now adopt a fixed cut-off date scheme. We will release twice a 561year. The upcoming $RELEASE_VERSION timeline are as followed: 562 563* Last posting date: $RELEASE_CUTOFF 564* Hard code freeze: $RELEASE_FREEZE 565* RC1: TBD 566* Release: $RELEASE_DATE 567 568Note that we don't have freeze exception scheme anymore. All patches 569that wish to go into $RELEASE_VERSION must be posted no later than the last posting 570date. All patches posted after that date will be automatically queued 571into next release. 572 573RCs will be arranged immediately after freeze. 574 575We recently introduced a jira instance to track all the tasks (not only big) 576for the project. See: https://xenproject.atlassian.net/projects/XEN/issues. 577 578Most of the tasks tracked by this e-mail also have a corresponding jira task 579referred by XEN-N. 580 581I have started to include the version number of series associated to each 582feature. Can each owner send an update on the version number if the series 583was posted upstream? 584 585= Projects = 586 587EOF 588 589POST=`mktemp` 590cat <<EOF > $POST 591 592EOF 593 594# Preamble 595cat $PRE 596rm $PRE 597# Body 598cat $FILE | awk -f $AWK_FILE 599rm $AWK_FILE 600rm $FILE 601cat $POST 602rm $POST 603``` 604