{"id":11540,"date":"2019-07-25T12:16:40","date_gmt":"2019-07-25T12:16:40","guid":{"rendered":"https:\/\/powerm.ma\/?p=11540"},"modified":"2019-07-25T12:20:30","modified_gmt":"2019-07-25T12:20:30","slug":"lessons-learned1","status":"publish","type":"post","link":"https:\/\/powerm.ma\/lessons-learned1\/","title":{"rendered":"Continuous operations : lessons learned from a simultaneous multiple disk failure issue on a virtualizing RAID computer data storage system\u00a0"},"content":{"rendered":"<h4 style=\"text-align: center;\"><span style=\"color: #008080;\">Continuous operations : lessons learned from a simultaneous multiple disk failure issue on a virtualizing RAID computer data storage system &#8211;\u00a0Published 25 July 2019 &#8211; ID PM00017 &#8211; 10 min read<\/span><\/h4>\n<h3>The Story<\/h3>\n<p>The customer experienced an outage on June 2019 which was triggered by a V7000 GEN2 disk related issue resulting in an offline mdisk :\u00a0mdisk offline with 3 failed drives with a potential loss of data.<\/p>\n<p><span style=\"color: #008080;\"><strong>Problem timeline:<\/strong><\/span><\/p>\n<ul>\n<li><strong>hh:mm<\/strong> the drive <strong><span style=\"color: #0000ff;\">ID1<\/span><\/strong> reported a self-test failure and was set to offline due to many reported errors. The RAID-5 Array mdisk0 rebuild started to drive <span style=\"color: #0000ff;\"><strong>ID2<\/strong><\/span>.<\/li>\n<li><strong>hh:mm+5<\/strong> the drive <strong><span style=\"color: #0000ff;\">ID3<\/span><\/strong> reported multiple hardware failures but stayed online allowing rebuild to drive <strong><span style=\"color: #0000ff;\">ID2<\/span><\/strong> to finish. When rebuild to drive <span style=\"color: #0000ff;\"><strong>ID2<\/strong><\/span> finished,the drive <span style=\"color: #0000ff;\"><strong>ID3<\/strong><\/span> was set failed due to excessive error and rebuild to drive <span style=\"color: #0000ff;\"><strong>ID4<\/strong><\/span> started.<\/li>\n<li><strong>hh+4:mm<\/strong>\u00a0the drive <span style=\"color: #0000ff;\"><strong>ID2<\/strong><\/span> went offline during the rebuild of drive <span style=\"color: #0000ff;\"><strong>ID3<\/strong><\/span> due to too many medium errors thus not allowing the rebuild to drive <span style=\"color: #0000ff;\"><strong>ID4<\/strong><\/span> to finish and setting array offline.<\/li>\n<\/ul>\n<p><span lang=\"EN-US\">The disks that failed with \u201cDrive reporting too many medium errors\u201d are related to a particular drive type (HUC156060CSS20 600 GB 15k).<\/span><\/p>\n<h4>Losing 3 Drives in a RAID5+ Hot Spare configuration looks like a <strong><span style=\"color: #0000ff;\">Final Destination<\/span><\/strong> movie scene \ud83d\ude00<\/h4>\n<h3>Overview of Hitachi King Cobra F drives\u00a0HUC156060CSS200<\/h3>\n<p>The Ultrastar C15K600, is\u00a0the world\u2019s fastest hard disk drive in a 15K RPM, 2.5-inch small form factor hard drive and ideally suited for mission-critical data center and high performance computing environments.<\/p>\n<p>The best-in-class performance is achieved through several innovations, including media caching technology that provides a large caching mechanism for incoming data resulting in significantly enhanced write performance.<\/p>\n<p>The C15K600 is HGST\u2019s first hard drive to leverage an industry-leading 12Gb\/s Serial-Attached SCSI (SAS) interface enabling very high transfer rates between host and drive, supporting the performance and reliability needed within the most demanding enterprise computing environments like online transaction processing (OLTP), big data analytics, multi-user applications and data warehousing.<\/p>\n<p>The drive is used on EMC VMAX,EMC VNX, EMC VNX2,HPE 3PAR and IBM Storwize.<\/p>\n<p><img loading=\"lazy\" class=\" wp-image-11547 aligncenter\" src=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/CobraDrive-300x232.png\" alt=\"\" width=\"407\" height=\"315\" srcset=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/CobraDrive-300x232.png 300w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/CobraDrive-768x594.png 768w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/CobraDrive-1024x791.png 1024w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/CobraDrive.png 1576w\" sizes=\"(max-width: 407px) 100vw, 407px\" \/><\/p>\n<h3>Talking Math: Mean Time To Data Loss (MTTDL)<\/h3>\n<p>Mean Time To Data Loss (MTTDL)<strong> is one<\/strong> of \u00a0the standard reliability metric in storage systems. MTTDL represents a simple formula that can be used to compare the reliability of small disk arrays and to perform comparative trending analyses.<\/p>\n<p>In the storage reliability community, the MTTDL is calculated using Continuous-time Markov Chains (a.k.a. Markov model).<\/p>\n<p><img loading=\"lazy\" class=\" wp-image-11556 aligncenter\" src=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/Markov-300x198.png\" alt=\"\" width=\"420\" height=\"277\" srcset=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/Markov-300x198.png 300w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/Markov-768x507.png 768w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/Markov-1024x677.png 1024w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/Markov.png 1922w\" sizes=\"(max-width: 420px) 100vw, 420px\" \/><\/p>\n<p>The last time I heard about Markov Chains was back in 2001 when i was studying <strong>Queueing<\/strong><b class=\"b5\">\u00a0Network S<\/b><b class=\"b4\">ys<\/b><b class=\"b3\">tem<\/b><b class=\"b2\">s<\/b>. One of the intriguing course, turns out handy in diagnosing one of the common issues in the storage infrastructure nowadays. It&#8217;s not a Math article, so let&#8217;s make it simple since we only need to gather two informations : the probability of read error\u00a0during rebuild and probability of data loss over time.<\/p>\n<p><img loading=\"lazy\" class=\" wp-image-11560 aligncenter\" src=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/RAID-MTTL-300x78.png\" alt=\"\" width=\"670\" height=\"174\" srcset=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/RAID-MTTL-300x78.png 300w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/RAID-MTTL-768x200.png 768w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/RAID-MTTL-1024x266.png 1024w\" sizes=\"(max-width: 670px) 100vw, 670px\" \/><\/p>\n<p>The odds of read error\u00a0during rebuild is 0.001055\u00a0and the odds of data loss over 5 years is\u00a00.000538\u00a0\ud83d\ude15 so what really happened ?<\/p>\n<h3>IBM Technology Support Services and EMC Technical Advisory knowledge base<\/h3>\n<p>Doing some research on the valuable EMC and IBM Support Knowledge base , we find some clue of what potentially make those drives offline:<\/p>\n<p class=\"pageType noSecondHeader\">EMC TA 195555: VNX, VNXe, Symmetrix VMAX, CLARiiON CX4 Series: Certain 600GB 15K RPM SAS and FC disk drives may <span style=\"color: #eb0e0e;\"><strong>experience increased replacement rates when drives remain idle for extended periods of time, or when unused space is allocated.<\/strong><\/span><\/p>\n<p>Dell EMC has determined that certain 600GB 15K RPM Serial Attached SCSI (SAS) and Fibre Channel (FC) disk drives may experience increased replacement rates when the drives have remained idle for extended periods of time.\u00a0<strong><span style=\"color: #eb0e0e;\">Affected drives may experience increased replacement rates when drives have remained idle for extended periods of time, which may lead to data unavailability.<\/span><\/strong><\/p>\n<p><img loading=\"lazy\" class=\" wp-image-11582 aligncenter\" src=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/EMC-ETA-300x172.png\" alt=\"\" width=\"443\" height=\"254\" srcset=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/EMC-ETA-300x172.png 300w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/EMC-ETA-768x441.png 768w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/EMC-ETA-1024x588.png 1024w\" sizes=\"(max-width: 443px) 100vw, 443px\" \/><\/p>\n<p>IBM notes: on some Hitachi King Cobra F drives used on the IBM Storwize V7000 GEN2, the drives can exhibit either a higher failure rate and\/or a very high number of medium errors on many drives and this can result in a higher risk of data loss.A large proportion of <strong><span style=\"color: #ff0000;\">Power-On-Idle Hours<\/span> <\/strong>further aggravates the issue. Indications are that if a drive has experienced a larger number of Power-On-Idle cycles then it will be much more likely to be exposed to this issue. <strong><span style=\"color: #ff0000;\">A typical case of error count of a drive occurs when it performs little I\/O for long periods and then performs heavy I\/O<\/span><\/strong>.<\/p>\n<p>Regarding our customers drives , the IBM L3 Support confirm the issue and recommend to upgrade <strong>the drives firmware to J2GF after converting the array to RAID6.<\/strong><\/p>\n<p><img loading=\"lazy\" class=\" wp-image-11585 aligncenter\" src=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/IBM-TSS-300x192.png\" alt=\"\" width=\"467\" height=\"299\" srcset=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/IBM-TSS-300x192.png 300w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/IBM-TSS-768x491.png 768w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/IBM-TSS-1024x654.png 1024w\" sizes=\"(max-width: 467px) 100vw, 467px\" \/><\/p>\n<p>Now we have a pretty much clear idea of what would be the root cause analysis of the three drives failures on our V7000, lets see how we were able to recover data.<\/p>\n<h3>The recovery process<\/h3>\n<p>Hard drive recovery is the process of recovering data and restoring a hard drive to its last known good configuration, after a system\/hard drive crashes or is corrupted\/damaged.<\/p>\n<p>Recovering data from physically damaged hardware can involve multiple techniques.\u00a0Some damage can be repaired by replacing parts in the hard disk. This alone may make the disk usable, but there may still be logical damage. <strong>A specialized disk-imaging procedure<\/strong> is used to recover every readable bit from the surface. Once this image is acquired and saved on a reliable medium, the image can be safely analyzed for logical damage and will possibly allow much of the original file system to be reconstructed.<\/p>\n<p>Having that said we hired<span lang=\"EN-US\">\u00a0a specialized repair company (Ontrack) to repair Disk <span style=\"color: #000000;\"><strong><span style=\"color: #0000ff;\">ID2<\/span><\/strong>. Due to the emergency of the situation, we sent the disk securely on a 3 hours flight to Paris.\u00a0<\/span><\/span><\/p>\n<p><span lang=\"EN-US\" style=\"color: #0000ff;\"><strong>48 hours later Ontrack inform us that they were able to copy 99.9% of sectors from the Disk ID2 to a new\u00a0<span style=\"caret-color: #333399;\">disk<\/span>\u00a0ID5 provided by IBM.<\/strong><\/span><\/p>\n<p>Back to Casablanca Customer Datacenter, we needed to insert the disk <strong><span style=\"color: #0000ff;\">ID5<\/span><\/strong> on the V7000 but we had two challenges:<\/p>\n<ul>\n<li>The V7000 is under IBM SVC.<\/li>\n<li>We need to manually mark disk <span style=\"color: #0000ff;\"><strong>ID5<\/strong><\/span> as <span style=\"color: #0000ff;\"><strong>ID2<\/strong><\/span> so the RAID group would be able to rebuild volumes.<\/li>\n<\/ul>\n<p><strong><span style=\"color: #008080;\">Following an excellent commitment and valuable implication of IBM Systems ,TSS , Support L3 \u00a0and Development team, IBM provided us with a crafted iFix pre tested in the same conditions as those of PowerM customer (V7000 Gen 2 with Cobra Drives and 7.6 firmware).<\/span><\/strong><\/p>\n<div>The ifix magic drive <strong><span style=\"color: #0000ff;\">ID5<\/span><\/strong> in place of\u00a0<span style=\"color: #0000ff;\"><strong>ID2<\/strong><\/span>. <strong>Drive <span style=\"color: #0000ff;\">ID5<\/span> had 99.9% of the data needed<\/strong> , \u00a0the areas which were made zeros on drive <span style=\"color: #0000ff;\"><strong>ID5<\/strong><\/span> end up as mangled data:<\/div>\n<div><\/div>\n<ul>\n<li>The ifix was installed in service mode, both nodes was put in service mode.<\/li>\n<li>Executed #satask installsoftware -file &lt;INSTALL_PACKAGE&gt;.<\/li>\n<li>When both nodes were back up #satask stopservice.<\/li>\n<li>Once cluster was back online ran #svctask recoverarray 1 \u00a0and array started rebuilding.<\/li>\n<\/ul>\n<p><img loading=\"lazy\" class=\" wp-image-11594 aligncenter\" src=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/iFix-300x127.png\" alt=\"\" width=\"538\" height=\"228\" srcset=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/iFix-300x127.png 300w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/iFix-768x326.png 768w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/iFix-1024x434.png 1024w\" sizes=\"(max-width: 538px) 100vw, 538px\" \/><\/p>\n<ul>\n<li>After some checks on the AIX LPAR box we ran #fsck to check file system consistency : nor file system inode map or block allocation map were corrupted.<\/li>\n<li>Ran Oracle Database procedure to recover (out of scope of this blog).<\/li>\n<\/ul>\n<h3>10 Lessons learned<\/h3>\n<h4>Deeply understand Continuous Availability vs HA and DR<\/h4>\n<p>High availability (HA) and disaster recovery (DR) are relatively mature approaches that are typically well-known even by non-technical people. Continuous availability is not as mature and can be confused with HA or DR. People still think in terms of \u201chow many 9s\u201d they can achieve (99.999% uptime, for example). But that is an HA topic.<\/p>\n<p>Continuous Availability(CA) = High Availability(HA) + Continuous Operations(CO)<\/p>\n<p><img loading=\"lazy\" class=\" wp-image-11233 aligncenter\" src=\"https:\/\/powerm.ma\/wp-content\/uploads\/2018\/07\/Capture-d\u2019\u00e9cran-2018-07-10-\u00e0-15.57.24-300x41.png\" alt=\"\" width=\"549\" height=\"75\" srcset=\"https:\/\/powerm.ma\/wp-content\/uploads\/2018\/07\/Capture-d\u2019\u00e9cran-2018-07-10-\u00e0-15.57.24-300x41.png 300w, https:\/\/powerm.ma\/wp-content\/uploads\/2018\/07\/Capture-d\u2019\u00e9cran-2018-07-10-\u00e0-15.57.24-768x104.png 768w, https:\/\/powerm.ma\/wp-content\/uploads\/2018\/07\/Capture-d\u2019\u00e9cran-2018-07-10-\u00e0-15.57.24-1024x139.png 1024w\" sizes=\"(max-width: 549px) 100vw, 549px\" \/><\/p>\n<p>To justify the cost of continuous availability, a measurable impact on the business must be calculated. But often, as soon as the outage is past or if a data center has been fairly stable, the business side forgets the critical need for continuous availability until the next outage occurs.<\/p>\n<p>Read more on \u00a0<a href=\"https:\/\/powerm.ma\/always-business-considerations-continuous-availability\/\">https:\/\/powerm.ma\/always-business-considerations-continuous-availability\/<\/a><\/p>\n<h4>Be realistic about the cost of continuous availability<\/h4>\n<p>The cost of implementing continuous availability initiatives can create a high hurdle.<\/p>\n<p>You customers and partners may have unrealistic expectations about implementing continuous availability given the price of hardware. Simply throwing hardware at the problem and building a custom solution is a recipe for high maintenance and unattainable SLAs. The reality is that continuous availability is much more than just hardware and software. The facility prerequisite, the process and resource requirements should be actively taken into consideration.<\/p>\n<h4>Challenge the partner and vendor support<\/h4>\n<p><strong><span style=\"color: #008080;\">ALWAYS<\/span><\/strong>\u00a0ensure that you have a back-to-back support contract with your hardware and middleware providers (IBM,EMC,Redhat,Vmware,Oracle,&#8230;).<strong><span style=\"color: #ff0000;\">There are always critical incidents that can be only resolved by the L3 and vendors development team.<\/span><\/strong><\/p>\n<p>When you are facing a potential loss of data and you are confident that you&#8217;ve correctly designed your infrastructure by respecting all known best practices, don&#8217;t accept answers from you partners and vendors like (even they are legitimate) :<\/p>\n<ul>\n<li>You should restore the last known good backup\/ replica.<\/li>\n<li>You lost two disks in a RAID5 array so you will certainly loose all data in the array.<\/li>\n<li>You didn&#8217;t apply the latest firmware.<\/li>\n<li>&#8230;<\/li>\n<\/ul>\n<p><strong><span style=\"color: #008080;\">There is always a way to recover all or some of the data and you should never give up until you try all possibilities.<\/span><\/strong><\/p>\n<h4>Monitor on daily basis<\/h4>\n<ul>\n<li>If a disk fails and you don&#8217;t detect it, it&#8217;s only a matter of time until its partner will go as well, hence we highly recommend using IBM Spectrum Control as an integrated data and storage management software that will provide you monitoring, automation and analytics for your storage systems including IBM Storwize, SVC and SAN.<\/li>\n<li>Set up Call Home, Email Alert and Inventory Configuration for your IBM storage.<\/li>\n<li>Subscribe yourself to IBM &#8216;My Notification&#8217; for your IBM product.<\/li>\n<\/ul>\n<h4>Mirror everything<\/h4>\n<p><strong><span style=\"color: #008080;\">You should act like a cloud provider: the data must be available regardless of how important it is or whom is using it !<\/span><\/strong> if they are some inhibitors (cost of replication line , additional storage arrays, datacenter footprint, facilities,&#8230;) you should discuss with stockholders and lower the company expectations.<\/p>\n<h4>Backup everything<\/h4>\n<p>With <strong>next-gen data backup and disaster recovery<\/strong> (reduction algorithms, compression , deduplication ,space efficiency , copy data management ,&#8230;) we highly recommend to backup not only the production but also every single VM, file systems or databases accessible by some users even occasionally, you only need to adjust your data protection policy and use world-class solutions like IBM Spectrum Protect, Protect Plus, EMC DataDomain,&#8230;<\/p>\n<h4>Consider a global tech refresh every 5 \u00a0to 6 years<\/h4>\n<p>The technology is moving very fast.You should consider a tech refresh every five to six year. Here is a simple case of the evolution of IBM SVC over the last fifteen years:<\/p>\n<p><img loading=\"lazy\" class=\" wp-image-11612 aligncenter\" src=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/SVC-RoadMap-300x59.png\" alt=\"\" width=\"570\" height=\"112\" srcset=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/SVC-RoadMap-300x59.png 300w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/SVC-RoadMap-768x150.png 768w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/SVC-RoadMap-1024x200.png 1024w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/SVC-RoadMap.png 2020w\" sizes=\"(max-width: 570px) 100vw, 570px\" \/><\/p>\n<div class=\"para \">According to Gartner ,by 2022, artificial intelligence (AI)\/machine learning will represent more than 25% of solid-state array (SSA) workloads and by 2020, <strong>30% of SSAs will be based on nonvolatile memory express (NVMe)<\/strong> technology.<\/div>\n<div>Solid-state arrays have become faster, smaller and more reliable, and they&#8217;re a safe business decision, due to guarantees that can&#8217;t be obtained in other areas of infrastructure.<strong>Plan to migrate your entire mission critical storage to All Flash and end-to-end NVMe by the end of 2020.<\/strong><\/div>\n<div><\/div>\n<h4>Upgrade to the latest recommended Firmware<\/h4>\n<p>Be sure that you have at least N-2 version of firmware applied to you storage systems, for instance your\u00a0IBM Storwize V7000 (2076) should be at 8.1 level. Review the firmware recommendation at least every 3 month.<\/p>\n<h4>Use RAID6\/DRAID over RAID5<\/h4>\n<p>When choosing a RAID type, it is imperative to think about its impact on disk performance and application IOPS.<\/p>\n<p>In a RAID 5 implementation, a write operation might manifest as four I\/O operations. When performing I\/Os to a disk configured with RAID 5, the controller must scan, calculate, and write a parity segment for every data write operation.<\/p>\n<p>In RAID 6, which maintains dual parity, a disk write requires three read operations: two parity and one data. After calculating both the new parities, the controller performs three write operations: two parity and an I\/O. Therefore, in a RAID 6 implementation, the controller performs six I\/O operations for every write I\/O, and therefore the write penalty\u00a0is 6.<\/p>\n<p><img loading=\"lazy\" class=\" wp-image-11565 aligncenter\" src=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/RAID5vsRAID6-300x41.png\" alt=\"\" width=\"665\" height=\"91\" srcset=\"https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/RAID5vsRAID6-300x41.png 300w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/RAID5vsRAID6-768x105.png 768w, https:\/\/powerm.ma\/wp-content\/uploads\/2019\/07\/RAID5vsRAID6-1024x140.png 1024w\" sizes=\"(max-width: 665px) 100vw, 665px\" \/><\/p>\n<p>How can we mitigate the RAID6 write penalty?<\/p>\n<ul>\n<li>With the advanced cache-based capabilities on the IBM Storwize V7000 and IBM SVC , the write-penalty associated with additional parity will not impact application performance:\u00a0the write response time is hidden from the hosts by the write cache.<\/li>\n<li>If you are using massive replication from a V7000 \u00a0to another system consider using additional CPU.<\/li>\n<li>When using large drives, consider implementing Distributed RAID :\n<ul>\n<li>Faster drive rebuild improves availability and enables use of lower cost larger drives with confidence<\/li>\n<li>All drives are active, which improves performance especially with flash drives<\/li>\n<li>No \u201cidle\u201d spare drives<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h4>Choose the right partner<\/h4>\n<p>Choose Power Maroc\u00a0\ud83d\ude0e<\/p>\n<h1>References<\/h1>\n<ul>\n<li>IBM Reliability of Data Storage Systems -Zurich Research Laboratory &#8211;\u00a0Keynote NexComm 2015<\/li>\n<li>Hard Disk Drive Specification Ultrastar C15K600 2.5\u201d SAS Hard Disk Drive<\/li>\n<li>Mean time to meaningless: MTTDL, Markov models, and storage system reliability<\/li>\n<li>Redbook Implementing the IBM Storwize V7000 Gen2 &#8211;\u00a0ISBN-10: 0738440264- SG24-8244-00<\/li>\n<li>Always On: Business Considerations for Continuous Availability IBM REDP5090<\/li>\n<li>EMC Technical Note TA 195555<\/li>\n<li>Critical Capabilities for Solid-State Arrays\u00a0Published 6 August 2018 &#8211; ID G00338538<\/li>\n<\/ul>\n<h2>Disclaimer<\/h2>\n<p><span lang=\"EN-US\">This content was provided for informational purposes only. The opinions and insights discussed are mine and do not necessarily represent those of the Power Maroc S.A.R.L.<span class=\"apple-converted-space\">\u00a0<\/span><\/span><\/p>\n<p><span lang=\"EN-US\">Nothing contained in this article is intended to, nor shall have the effect of, creating any warranties or representations from Power Maroc S.A.R.L or its Partners (particularly IBM and DELL Technologies), or altering the terms and conditions of any agreement you have with Power Maroc S.A.R.L.<span class=\"apple-converted-space\">\u00a0<\/span><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Continuous operations : lessons learned from a Multiple Disk Failure Issue on a Virtualizing RAID Computer Data Storage System<\/p>\n","protected":false},"author":2,"featured_media":11648,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[245],"tags":[267,188,265,80,85,266,84,270,260,276,98,246,250,255,83,259,261,263,273,275,257,256,277,279,264,271,221,274,252,253,86,211,268,102,269,278,254,251,262,258,247,249,272,248],"_links":{"self":[{"href":"https:\/\/powerm.ma\/wp-json\/wp\/v2\/posts\/11540"}],"collection":[{"href":"https:\/\/powerm.ma\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/powerm.ma\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/powerm.ma\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/powerm.ma\/wp-json\/wp\/v2\/comments?post=11540"}],"version-history":[{"count":96,"href":"https:\/\/powerm.ma\/wp-json\/wp\/v2\/posts\/11540\/revisions"}],"predecessor-version":[{"id":11651,"href":"https:\/\/powerm.ma\/wp-json\/wp\/v2\/posts\/11540\/revisions\/11651"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/powerm.ma\/wp-json\/wp\/v2\/media\/11648"}],"wp:attachment":[{"href":"https:\/\/powerm.ma\/wp-json\/wp\/v2\/media?parent=11540"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/powerm.ma\/wp-json\/wp\/v2\/categories?post=11540"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/powerm.ma\/wp-json\/wp\/v2\/tags?post=11540"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}